Information-Regularized Attention for Visual-Centric Reasoning

Vision–language models (VLMs) have become a paradigmfor multimodal learning, yet remain unstable due to object hallucination,weak visual grounding, and catastrophic forgetting after full-parameterinstruction tuning. We claim these failures result from a lack of explicitcontrol over visual representation learning during the standard next-token prediction objective. As a result, visual embeddings thus becomepassively optimized and prone to injecting redundant or spurious signals.To counter this, we introduce Information-Regularized Attention (IRA),a stochastic attention mechanism that explicitly regulates the amount ofvisual information injected into the hidden states of intermediate trans-former layers. This local reparameterization translates uncertainty aboutvisual representations into local noise that is independent across datapoints. Beyond evaluating model performance, we also quantify embed-ding properties, where IRA produces smoother curvature trajectoriesand suppresses attention-sink across all layers, indicating a more stabletransformation of the visual signal. Our results suggest that stochasticattention is not merely a regularizer but a key contributor to represen-tation learning in a generative architecture, offering a new direction forbuilding more reliable VLMs.