Optimization-Guided Diffusion for Interactive Scene Generation

Realistic and diverse multi-agent driving scenes are crucial for evalu-ating autonomous vehicles, but safety-critical events which are essential for thistask are rare and underrepresented in driving datasets. Data-driven scene genera-tion offers a low-cost alternative by synthesizing complex traffic behaviors fromexisting driving logs. However, existing models often lack controllability or yieldsamples that violate physical or social constraints, limiting their usability. Wepresent OMEGA, an optimization-guided, training-free framework that enforcesstructural consistency and interaction awareness during diffusion-based samplingfrom a scene generation model. OMEGA re-anchors each reverse diffusion stepvia constrained optimization, steering the generation towards physically plausibleand behaviorally coherent trajectories. Building on this framework, we formu-late ego–attacker interactions as a game-theoretic optimization in the distributionspace, approximating Nash equilibria to generate realistic, safety-critical adver-sarial scenarios. Experiments on nuPlan and Waymo show that OMEGA improvesgeneration realism, consistency, and controllability, increasing the ratio of physi-cally and behaviorally valid scenes from 32.35% to 72.27% for free explorationcapabilities, and from 11% to 80% for controllability-focused generation. Our ap-proach can also generate 5× more near-collision frames with a time-to-collisionunder three seconds while maintaining the overall scene realism.