Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding

Despite the remarkable capabilities of Multimodal Large Lan-guage Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishesas the text sequence lengthens, leading to text generation detached fromvisual constraints. We attribute this degradation to the inherent induc-tive bias of Multimodal RoPE, which penalizes inter-modal attention asthe distance between visual and text tokens increases. To address this, wepropose inter-modal Distance Invariant Position Encoding (DIPE), asimple but effective mechanism that disentangles position encoding basedon modality interactions. DIPE retains the natural relative positioningfor intra-modal interactions to preserve local structure, while enforc-ing an anchored perceptual proximity for inter-modal interactions. Thisstrategy effectively mitigates the inter-modal distance-based penalty, en-suring that visual signals remain perceptually consistent regardless ofthe context length. Experimental results demonstrate that by integrat-ing DIPE with Multimodal RoPE, the model maintains stable visualgrounding in long-context scenarios, significantly alleviating visual fad-ing while preserving performance on standard short-context benchmarks.Code is available at https://github.com/lchen1019/DIPE.