QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

The performance of vision-language models (VLMs) in videounderstanding declines with increasing video duration, as video momentsunrelated to the query confuse their language components. Multimodalretrieval has emerged as a critical component of video understanding,addressing this challenge by localizing key visual evidence. However, ex-isting multimodal retrieval methods suffer from biased relevance estima-tion, limited diversity, and temporal collapse. In this paper, we proposeQSVideo, a unified framework that systematically addresses relevance,diversity, and temporal modeling in video retrieval. We first introducea query-conditioned semantic ranker, QSRanker, which reformulates ar-bitrary questions into retrieval-friendly queries and estimates structuredrelevance along object, action, and location dimensions. Building uponthis, we design QSRetrieval to jointly optimize relevance and diversityfor more informative frame selection. Moreover, we propose temporalalignment strategies tailored for both long and streaming videos to im-prove evidence recall. Extensive experiments on long and streaming videobenchmarks demonstrate that QSVideo greatly enhances video VLM per-formance under strict frame limit constraints. The code is available athttps://github.com/human-analysis/QSVideo.