AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning

Large vision-language models incur substantial inference costsbecause high-resolution inputs introduce thousands of visual tokens, manyof which are redundant for a given query. Existing pruning methods of-ten combine query relevance and token diversity, yet these objectivescan conflict under aggressive compression: relevance-driven selection mayoverconcentrate the budget on correlated local evidence, while diversity-driven selection may suppress indispensable tokens or retain distinctbut uninformative regions. We introduce AnchorPrune, a training-freeframework that first constructs a protected relevance anchor and thenexpands it with complementary visual context. AnchorPrune adaptivelydetermines the anchor size from the novelty profile of relevance-rankedtokens, preserving a compact set of query-critical evidence, and allocatesthe remaining budget through importance-weighted novelty to recover in-formative, non-redundant context relative to the anchor. This ordered de-sign prevents contextual expansion from displacing indispensable querycues while improving overall visual coverage. AnchorPrune is lightweight,architecture-aware, and requires neither retraining nor model modifica-tion. Across image and video vision-language models and benchmarks, itconsistently improves the accuracy–efficiency trade-off over training-freebaselines, particularly under severe compression. On LLaVA-NeXT-7B,AnchorPrune preserves 97.6% of full-token performance using only 160of 2,880 visual tokens. These results establish relevance-anchored contex-tual expansion as an effective principle for efficient multimodal inference.Code is available at https://github.com/MULTI-cau/AnchorPrune.