Dense Video Understanding with Inter-tokenization Acceleration

High temporal resolution is essential for capturing fine-grained dynamics in video understanding, yet current video large language models (VLLMs) and benchmarks predominantly rely on low-frame-rate sampling, discarding dense temporal information. This design compromise is driven by the high cost of frame-wise tokenization, which incurs redundant computation and linear growth in token count with video length. While effective for slowly changing content, such engineering trade-offs severely limit tasks where information is distributed across nearly every frame, such as educational or high-motion videos that require frameby-frame reasoning. To address this gap, we introduce the novel task of High-FPS Video Understanding, the first task specifically designed to evaluate video comprehension at high frame rates, where information is present in nearly every frame. We further propose the first benchmark, DIVE-Bench (Dense Information Video Evaluation Benchmark), to expose the limitations of existing benchmarks whose question– answer pairs are not sensitive to dense temporal variations. To enable efficient high-FPS processing, we propose Gated Residual Tokenization (GRT), a codec-style inter-tokenization framework that operates not only after but also during tokenization. First, Motion-Compensated Gated Inter-Tokenization applies pixel-level motion estimation to gate static regions during tokenization, achieving sub-linear growth in both tokenization time and token count. Second, Semantic-Scene Token Merging further reduces redundancy by merging semantically similar static tokens while preserving dynamic content. Extensive experiments on DIVEBench show that GRT improves high-FPS efficiency while preserving or improving answer quality under dense temporal perception tasks. These results highlight the importance of dense temporal modeling and establish codec-style inter-tokenization as a practical direction for scalable high-FPS video understanding.