DIVA: Instruction-Aware Vision Token Pruning via Dual-Probe Attention Discrepancy

Large vision–language models (LVLMs) incur substantial com-putational and memory overhead due to quadratic attention complexityand enlarged key–value (KV) caches. Training-free vision token prun-ing is promising, yet most methods rely on text-to-vision attention-scoremagnitudes, which often entangle instruction-relevant signals with po-sitional bias and visual attention sinks. To address this issue, we pro-pose DIVA, an instruction-aware, layer-wise vision token pruning methodbased on dual-probe attention discrepancy, with two key modules: DIVA-Pruner and VF-Calibrator. DIVA-Pruner selects important tokens us-ing the discrepancy between attention distributions of contextual andnon-contextual probe sets. A lightweight VF-Calibrator calibrates unim-portant tokens that bypass self-attention, mitigating representation mis-match and enabling effective reuse in subsequent LVLM blocks. On di-verse benchmarks and LVLM baselines, DIVA consistently outperformsprior methods across token budgets, even in highly constrained regimes.Notably, under a 97.2% token reduction, DIVA retains 92.6% of baselineperformance on LLaVA-NeXT-7B with a 10.8× reduction in KV cache.