ROSE: Real-Time Open-World Scene Understanding from Monocular Video via Compact Multimodal 4D Scene Graphs

Vision-Language Models (VLMs) have advanced rapidly invisual and semantic understanding, yet they remain weak at dynamic 4Dspatial grounding in video. On spatio-temporal reasoning benchmarks,raw-video input often yields low and variable accuracy. 4D Scene Graphs(4DSGs) can supply missing structure, but two interface gaps remain:identity facets (appearance, geometry, motion) are often encoded throughlanguage-side descriptors rather than modality-separated inputs, andmost training targets static spatial relations instead of dynamic 4D at-tributes such as trajectories and velocity. We introduce Visual Spatio-Temporal Anchor (VISTA), a multimodal node encoding that representseach tracked object as a persistent, queryable entity, delegating appear-ance to anchor crops processed by the VLM’s vision encoder and geome-try and motion to explicitly structured tokens. We introduce Real-TimeOpen-World Scene Understanding (ROSE), a training-free monocularpipeline (single GPU) that runs at 10 Hz. Beyond inference-time prompt-ing, the 4DSGs produced by ROSE can serve as structured training-timecontext: fine-tuning a VLM on 4DSG-augmented data teaches it to usetrajectory and velocity tokens for dynamic reasoning, yielding a new stateof the art on DSR-Bench (67.5, +8.6 over the previous best), with gainsacross all 13 subtasks. ROSE achieves competitive accuracy across allthree benchmarks under real-time, single-GPU monocular constraints.