Dive into the implicit biases of low-rank vision-language alignment

Vision-language alignment—the stage that bridges pretrainedvision encoders and large language models—is widely treated as a formof pretraining requiring full-parameter updates. We challenge this viewand investigate what happens when low-rank adaptation is applied to theLLM during this stage instead. We find that low-rank alignment not onlyreduces computational costs but also outperforms full-parameter align-ment on most benchmarks. To understand this phenomenon, we system-atically characterize the implicit biases introduced by low-rank adap-tation during alignment. Empirically, we find that low-rank alignmentshifts model behavior from hallucinatory to conservative and preservesper-token linear separability of visual features that full-parameter align-ment disrupts—a phenomenon we term LS-curse. Geometrically, low-rank aligned models exhibit more homogeneous and structurally stablevisual representations, maintaining modality-specific knowledge ratherthan prematurely fusing entity-level semantics. Theoretically, we estab-lish two theorems showing that low-rank alignment induces preferencesfor parameter subspaces with flat gradients and feature subspaces robustto perturbations, providing a principled explanation for the observedstructure-preserving behavior. Extensive experiments cover ablation over100 alignment configurations, three families of low-rank operators, andvarious rank, encoder, and other settings.