Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs

Video Large Language Models (Video LLMs) incur high in-ference latency due to a large number of visual tokens provided to LLMs.To address this, training-free visual token pruning has emerged as asolution to reduce computational costs; however, existing methods areprimarily validated on Multiple-Choice Question Answering (MCQA)benchmarks, where coarse-grained cues often suffice. In this work, wereveal that these methods suffer a sharp performance collapse on fine-grained understanding tasks requiring precise visual grounding, such ashallucination evaluation. To explore this gap, we conduct a systematicanalysis and identify sink tokens–semantically uninformative tokens thatattract excessive attention–as a key obstacle to fine-grained video un-derstanding. When these sink tokens survive pruning, they distort themodel’s visual evidence and hinder fine-grained understanding. Moti-vated by these insights, we propose Sink-Token-aware Pruning (SToP),a simple yet effective plug-and-play method that introduces a sink scoreto quantify each token’s tendency to behave as a sink and applies thisscore to existing spatial and temporal pruning methods to suppress them,thereby enhancing video understanding. To validate the effectivenessof SToP, we apply it to state-of-the-art pruning methods (VisionZip,FastVid, and Holitom) and evaluate it across diverse benchmarks cover-ing hallucination, open-ended generation, compositional reasoning, andMCQA. Our results demonstrate that SToP significantly boosts perfor-mance, even when pruning up to 90% of visual tokens. Our code is avail-able at https://github.com/rlqja1107/SToP