Who Does What and Where to Go: Orthogonal Alignment and Hierarchical Planning for Multi-Entity Trajectories

Generating precise multi-entity 3D trajectories is fundamen-tal for interactive video generation and embodied world models. However,existing text-to-trajectory mappings face two bottlenecks: severe featureleakage from coupled multi-modal inputs, and physical collisions causedby blind end-to-end 3D coordinate regression. We propose ProgTraj-Director, a framework driven by physical property disentanglement andhierarchical spatial planning. To resolve the “who does what” attribu-tion ambiguity, our Structured Vision-Language Alignment (StructVLA)module disentangles static identity masks from dynamic spatial motionflows, binding them via orthogonal factorized tensor fusion in the latentspace. This formulation encourages entity-specific semantic separationand mitigates cross-entity feature leakage. To bridge the “where to go”semantic-physical gap, we formulate trajectory generation as hierarchicalspatial planning. The model first constructs a Geometry-Aware Bird’s-EyeView (BEV) to resolve anti-collision topological boundaries. Guided bythis structural prior, 3D trajectories are derived via progressive spatialdenoising to improve geometric and temporal consistency. To supportthis research, we introduce Stepwise-ME, a large-scale dataset of over31,000 interactive video clips (8M frames) with fine-grained stepwiseannotations. Extensive experiments show that our approach enhancesaction attribution and spatial coherence, providing a reliable structuralprior for downstream video generation. Code and dataset are available athttps://github.com/wzbos-token/Who-Does-What-and-Where-to-Go.