Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation
Abstract
Real-world deployment of embodied agents requires activeexploration, visual grounding, and interactive intent disambiguation. How-ever, existing frameworks often rely on privileged simulator states or as-sume complete instructions, bypassing realistic deployment challenges.To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent en-vironment APIs without oracle perception and integrates a simulateduser to enable human-in-the-loop interaction. Within this environment,we design diverse task compositions to drive data collection, supervisedfine-tuning, and online reinforcement learning, systematically optimizingagent performance. To comprehensively evaluate this approach, we in-troduce REAL-Bench, a benchmark spanning 241 tasks across activeexploration, visual distraction, articulated manipulation, and interactivedisambiguation. Experimental results demonstrate that our trained agentoutperforms leading commercial closed-source VLMs on interactive taskswith a 56.9% success rate. Further empirical analysis reveals that ourhierarchical training pipeline successfully aligns the model’s tool-use ca-pabilities while maintaining robust open-vocabulary reasoning under ex-tended exploration horizons. Finally, we deploy and evaluate our frame-work on a physical dual-arm mobile robot, where it achieves a 78.3%end-to-end success rate over 60 real-world episodes. These physical trialsdemonstrate robust zero-shot transferability to unseen household scenar-ios, validating that our sim-to-real-consistent design successfully bridgesthe reality gap for long-horizon mobile manipulation. Code is availableat https://github.com/InternRobotics/REAL.