BEVOpen3D: Towards Open-World 3D Object Detection in Bird's-Eye-View

The reliance on extensively annotated 3D data confines cur-rent autonomous driving systems to a closed-set detection paradigm, lim-iting their robustness in real-world open-world scenarios where novel ob-jects frequently emerge. To bridge this gap, we propose BEVOpen3D,a novel distillation framework that facilitates open-world 3D object de-tection by effectively transferring knowledge from 2D vision-languagemodels into the 3D domain via the Bird’s-Eye-View (BEV) space undera partial-label setting. Our approach first generates initial 3D proposalsfor both seen and unseen categories using an open-vocabulary 2D detec-tor combined with a greedy spatial search strategy. We then introducea Triple-Source Label Refinement mechanism, which fuses original, lo-cally rectified (via a Seen-Guided Local Query), and globally re-examinedproposals to produce high-quality pseudo labels for unseen-category. Fi-nally, we propose a Heatmap Proposal Distillation strategy, where animage-based BEV teacher transfers its vision-grounded semantic priorsto a LiDAR-based student through aligned BEV heatmaps, enabling thestudent to detect both seen and unseen objects without requiring addi-tional 3D annotations. Extensive experiments validate the effectivenessof BEVOpen3D in open-world 3D detection.