S3-Prune: Stability-Aware Token Budgeting for Long-Form Video-Language Models

The deployment of long-form video-language models (Video-VLMs) is fundamentally constrained by the prohibitive scaling of atten-tion complexity and KV-cache overhead as temporal and spatial res-olutions increase. Existing pruning methods rely on local frame-levelheuristics, which fail to capture shifting information density and oftenresult in cumulative budget misallocation over long horizons. To addressthese challenges, we propose S3 -prune, a stability-aware token budgetingframework that jointly models spatio-temporal demand while ensuringconsistent allocation over time. S3 -prune characterizes token demand byintegrating spatial uncertainty (SU) from patch-level embedding shiftsand segment transition (ST) from inter-segment semantic variations. Tomitigate temporal instability, we introduce stability accumulation (SA)via a Kalman-filtered latent estimator, which smoothens stochastic spikesand provides uncertainty-aware margins for adaptive budgeting. Guidedby these stabilized budgets, S3 -prune executes a hierarchical two-stageselection process. Extensive evaluations across seven benchmarks andvarious backbones demonstrate that S3 -prune significantly advances theefficiency-accuracy frontier, reducing inference latency while remainingrobust even under aggressive token retention settings.