Multi-View Foundation Models

Foundation models are vital tools in various Computer Vision applications. They take as input a single RGB image and output a deep feature representation that is useful for various applications. However, in case we have multiple views of the same 3D scene, they operate on each image independently and do not always produce consistent features for the same 3D point. We propose a way to lift a Foundation Model into a Multi-View Foundation Model. Such a model takes as input a set of images and outputs a feature map for each image such that the features of corresponding points are as consistent as possible. This approach bypasses the need to build a consistent 3D model of the features. Specifically, we show how to augment any Transformer-based foundation model (i.e., DINO, SAM, CLIP) with intermediate 3D-attention layers that help match features across different views. Quantitative evaluations demonstrate that our approach achieves superior 3D-awareness while best preserving the native semantic space of the original foundation model, outperforming existing lifting techniques.