LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

Recent years have witnessed remarkable progress in autonomousdriving, yet generalization to long-tail and open-world scenarios remainsthe primary bottleneck for large-scale deployment. To address this, oneline of research explores LLMs and VLMs for their vision-language un-derstanding and reasoning capabilities, equipping AVs with the abilityto interpret rare and safety-critical situations when generating drivingactions. In parallel, another line investigates generative world models tocapture the spatio-temporal evolution of driving scenes, enabling agentsto imagine and evaluate possible futures before acting. Inspired by hu-man intelligence, which seamlessly unites understanding and imaginationas a hallmark of AGI, this work explores a unified model that bringsthese two capabilities together for autonomous driving. We present LM-GenDrive, the first framework that unifies LLM-based multimodal un-derstanding with generative world models for end-to-end closed-loop au-tonomous driving. Given multi-view camera inputs and natural-languageinstructions, our model generates both realistic future driving videosand corresponding control signals. By coupling an LLM with genera-tive video capabilities, LMGenDrive gains complementary benefits: fu-ture video prediction enhances spatio-temporal scene modeling, while theLLM provides strong semantic priors and instruction grounding learnedfrom large-scale pretraining. A progressive three-stage training strat-egy—ranging from vision pretraining to multi-step long-horizon driv-ing—is proposed to further improve stability and performance. The re-sulting model can also operate in two complementary modes: low-latencyonline planning and autoregressive offline video generation. Experimentsshow that LMGenDrive significantly outperforms state-of-the-art meth-ods on challenging closed-loop driving benchmarks, improving instruc-tion following, spatio-temporal understanding, and robustness to rarescenarios. Our work not only sets a new state-of-the-art in autonomousdriving, but also demonstrates that unifying multimodal understandingand generation provides a promising direction for building more gener-alizable and robust embodied decision-making systems.