Towards Geometry-Grounded Dense Semantic Matching with VGGT Priors

Semantic matching aims to establish pixel-level correspon-dences between instances of the same category, and previous approachesbased on 2D foundation models have achieved promising results on thistask. However, they fail to achieve geometry-grounded dense semanticmatching, which encompasses geometric awareness, manifold preserva-tion, and cross-image invisibility reasoning. This new matching task si-multaneously needs geometry-aware descriptors and holistic dense match-ing mechanisms, but existing approaches rarely provide both. To bridgethis gap, we leverage VGGT, a 3D geometric foundation model that in-herently provides strong geometry-grounded priors to support both capa-bilities in a unified framework. However, directly transferring faces twochallenges: task heterogeneity, where VGGT originally matches cross-view images of the same instance, not cross-instance variations in shapeor appearance; and scarce dense matching annotations for task adapta-tion. To address these challenges, we propose an approach that (i) retainsVGGT’s intrinsic capabilities by reusing early feature stages, fine-tuninglater ones, and adding a semantic head for bidirectional correspondences;and (ii) adapts VGGT for semantic matching under data scarcity throughcycle-consistent training strategy, synthetic data augmentation, and pro-gressive training recipe. Extensive experiments demonstrate that our ap-proach achieves superior geometric awareness, manifold preservation, andmatching reliability, outperforming previous baselines.