Agentic Collaborative Cognition for Zero-Shot 3D Understanding

Recent advancements have explored agentic zero-shot 3D un-derstanding by reformulating it as video keyframe understanding withMultimodal Large Language Models (MLLMs). However, existing meth-ods face an intrinsic bottleneck due to the finite observation perspectivesinherent in videos and the implicit perception of 3D scenes. In this pa-per, we propose a collaborative multi-agent framework that assigns aPlanning Agent to handle high-level viewpoint planning and supplementnovel perspectives, and a Perception Agent to explicitly summarize the3D scene into a structured holistic cognitive map. Specifically, PlanningAgent first analyzes this cognitive map to determine query-relevant view-points and supplements missing critical perspectives to ensure compre-hensive observation. Subsequently, Perception Agent documents object-level attributes from these views by assigning consistent instance identi-fiers across viewpoints, thereby integrating fragmented observations intothe holistic cognitive map. In parallel, it provides feedback to filter outmismatched candidate objects and guide subsequent viewpoint planning.Through this closed-loop iterative process, two agents collaboratively fig-ure out candidates until Perception Agent determines that sufficient in-formation has been captured to complete the task. Extensive experimentsdemonstrate that our method achieves state-of-the-art performance on6 benchmarks, with improvements of 11.1% Acc@0.5 on ScanRefer, 14.6BLEU-1 on 3D-assisted dialog, and 2.1 EM on SQA3D. Project Page:https://zhangbo135.github.io/agentic-collaborative-cognition/