MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein

Multimodal Large Language Models (MLLMs) inherit richrelational priors from their language backbones, yet often fail when askedto apply these relationships in visual contexts. We trace this failure to astructural blind spot: projection-based alignment trains each visual to-ken to carry the right semantics, but never asks whether the relationshipsbetween concepts survive the crossing from language to vision. To ad-dress this, we propose MIRROR (Mapping Inter-concept Relationsfrom language to visual Representation via Optimal-transport-basedRegularization), a geometric regularization framework that transfers re-lational priors from language to vision by exploiting the rich relationalstructure encoded in language representations. Specifically, we derive asurrogate loss from the proposed Semi-Inverse Gromov–Wasserstein(SI-GW) problem, an inverse geometric problem that aligns visual rep-resentations with language-derived relational priors. We show that thisformulation admits a unique closed-form solution that prescribes theideal visual relational structure implied by language geometry and cross-modal coupling. The structure of the formulation also enables efficientcomputation, making it applicable to long token sequences. ApplyingSI-GW inside decoder-only Transformers requires careful design. We in-troduce targeted strategies at the layer, head, and token levels to ensurestable extraction without additional parameters or inference cost. MIR-ROR improves relational consistency while preserving performance ongeneral vision-language tasks.