Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

Bird’s-eye view (BEV) perception fuses multi-camera imagesinto a unified top-down representation for autonomous driving. Despiterecent progress, state-of-the-art methods remain confined to closed-setscenarios, making them vulnerable to unpredictable real-world environ-ments. In this work, we introduce open-vocabulary BEV segmentation(OVBS), which leverages vision-language models (VLMs) to recognizecategories beyond the training set while maintaining precise BEV percep-tion and real-time efficiency. A key challenge in OVBS lies in the 3D geo-metric inconsistency inherent in the ill-posed lifting of 2D VLM semanticsinto BEV. To address this, we propose OVBEVSeg, a geometry-awareOVBS framework that enhances efficient Gaussian splatting (GS)-basedunprojection by leveraging robust 3D geometric constraints across threeprogressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D pro-jection for OV generalization; (2) joint 2D–BEV per-scene optimizationwith BEV structural constraints for 3D geometric consistency; and (3)3D geometric distillation for online efficiency. On the nuScenes dataset,OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even withno novel-class ground-truth labels, it remains competitive with self- andsemi-supervised baselines trained with up to 40% of ground-truth anno-tations. Furthermore, it achieves 2.5× faster inference with only 0.22×the memory consumption of projection-based methods.