Skip to yearly menu bar Skip to main content


Poster

Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding

Ruihuang Li · Zhengqiang ZHANG · Chenhang He · Zhiyuan Ma · Vishal Patel · Lei Zhang

Strong blind review: This paper was not made available on public preprint services during the review process Strong Double Blind
[ ]
Wed 2 Oct 1:30 a.m. PDT — 3:30 a.m. PDT

Abstract:

Recent vision-language pre-training models have exhibited remarkable generalization ability in zero-shot recognition tasks. However, their applications to 3D dense prediction tasks often encounter the difficulties of limited high-quality and densely-annotated 3D data. Previous open-vocabulary 3D scene understanding methods mostly focus on training 3D models using either image or text supervision while neglecting the collective strength of all modalities. In this work, we propose a Dense Multimodal Alignment (DMA) framework to densely co-embed different modalities into a common space for maximizing their synergistic benefits. Instead of extracting coarse view- or region-level text prompts, we leverage large vision-language models to extract complete category information and scalable scene descriptions to build the text modality, and take image modality as the bridge to build dense point-pixel-text associations. Besides, in order to enhance the generalization ability of the 2D model for downstream 3D tasks without compromising the open-vocabulary capability, we employ a dual-path integration approach to combine frozen CLIP visual features and learnable mask features. Extensive experiments show that our DMA method produces highly competitive open-vocabulary segmentation performance on various indoor and outdoor tasks.

Live content is unavailable. Log in and register to view live content