SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

Despite the remarkable success of large-scale pre-trained im-age representation models (i.e., vision encoders) across various visiontasks, they are predominantly trained on 2D image data and thereforeoften fail to capture 3D spatial relationships between objects and back-grounds in the real world, constraining their effectiveness in many down-stream applications. To address this, we propose SpatialBoost, a scalableframework that enhances the spatial awareness of existing pre-trained vi-sion encoders by injecting 3D spatial knowledge expressed in linguisticdescriptions. The core idea involves converting dense 3D spatial infor-mation from 2D images into linguistic expressions, which is then usedto inject such spatial knowledge into vision encoders through a LargeLanguage Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates densespatial knowledge and builds hierarchical spatial understanding. To val-idate effectiveness, we adapt SpatialBoost to state-of-the-art vision en-coders such as DINOv3, and evaluate its performance gains on a widerange of benchmarks requiring both 3D perception and general visionabilities. For instance, SpatialBoost improves DINOv3 performance from55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performancewith 3.8%p gain over the pre-trained DINOv3. Project page.