Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

Recently, open-vocabulary 3D object detection (3D-OVD)has gained increasing attention for its ability to detect unseen objects in3D scenes. Existing approaches typically adopt a two-stage pipeline thatfirst discovers novel objects using foundation models and then trains a3D-OVD model based on these discovered objects. Although effective,this pipeline often suffers from inaccurate localization and mismatchedclassification during the discovery stage, which subsequently limits theperformance of the model training stage. To address these limitations,we advocate for improving both the reliability of novel object discov-ery and the robustness of model training, and propose an innovativeframework. Specifically, for reliable discovery, our co-distillation strategydistills high-quality novel objects by applying Hungarian matching overa comprehensive score that incorporates geometric consistency, struc-tural objectness, and semantic certainty. To enhance robust model train-ing, we further propose a dual-guidance learning scheme, incorporatinga scene-awareness-guided uncertainty regularization for the regressionhead and an LLM-guided hierarchical alignment for the classificationhead, effectively mitigating the negative effects of imprecise 3D boundingboxes and semantic ambiguity. Extensive experiments on SUN RGB-Dand ScanNetV2 demonstrate that our method achieves significant per-formance gains over state-of-the-art approaches. Code is available athttps://github.com/shangboyuan/Co-3DGT.