ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance

Visual token pruning, which aims to compress and pruneredundant visual tokens, plays a critical role in efficient inference withlarge vision-language models (LVLMs). However, existing methods failto disentangle intra-modal visual redundancy from cross-modal redun-dancy between vision and language. We show that visual token diversityand task-specific token relevance are two crucial yet orthogonal factorsthat complement each other in conveying useful information and shouldtherefore be treated separately for more effective visual token pruning.Building upon this insight, we design ToDRE, a two-stage and training-free framework that incorporates Token Diversity and task RElevancefor effective token compression and efficient LVLM inference. Instead ofpruning redundant tokens, we introduce a greedy max-sum diversificationalgorithm that selects and retains a subset of diverse and representativevisual tokens after the vision encoder. On top of that, ToDRE leveragesan “information migration” mechanism to eliminate task-irrelevant visualtokens within certain decoder layers of the large language model (LLM),further improving token pruning and LVLM inference. Extensive exper-iments show that ToDRE prunes 90% of visual tokens after the visionencoder as well as all visual tokens in certain LLM decoder layers, leadingto a 2.6× speed-up in total inference time while maintaining 95.0% modelperformance plus excellent model compatibility.