Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Grounding

Video Temporal Grounding (VTG) localizes the temporalboundaries of query-relevant moments in long, untrimmed videos, mak-ing video-language-model prohibitively expensive. While recent training-free token pruning has shown success in video question answering, naivelyapplying these objectives to VTG causes drastic degradation, as VTGcrucially depends on boundary-sensitive evidence and cross-frame rea-soning chains. We therefore identify two VTG-specific pruning principles:evidence retention, which keeps query-critical patches especially aroundevent boundaries, and connectivity strength, which preserves cross-frameconnectivity for long-range evidence aggregation. Building on these in-sights, we propose SemVID, a training-free pruning framework that con-structs a compact yet coherent token subset with complementary seman-tic roles. SemVID first allocates per-frame budgets by balancing queryrelevance and inter-frame variation to avoid over-pruned segments, andthen selects three types of tokens: object tokens for diverse query-criticalevidence, motion tokens to capture meaningful transitions and serve ascross-frame relays, and context tokens for scene continuity. Extensive ex-periments show that SemVID achieves a strong accuracy-efficiency trade-off, retaining up to 95.4% mIoU with only 12.5% visual tokens and de-livering up to a 5.8× prefill speedup, consistently outperforming priormethods under the same budgets. Our code is available here.