Incentivizing Vision Language Models to Search for Long Video Question Answering

We introduce VSeek, an agentic framework that transformslong-video question answering (LVQA) from a passive, single-pass per-ception task into a multi-turn retrieval process. VSeek utilizes a naturallanguage-driven search to identify relevant context within long videosand is post-trained with reinforcement learning (RL) to jointly formulatetargeted search queries and reason over retrieved clips for LVQA. WhileRL post-training has revolutionized reasoning in symbolic domains suchas mathematics and code, its application to long-video understandingremains hindered by a lack of verified rewards. To ensure that the re-trieved context is relevant, we propose a novel neuro-symbolic approachthat bridges open-ended natural language with discrete visual verifica-tion. Specifically, complex user queries are compiled into formal tempo-ral logic specifications for systematically decomposing natural languagequestions into a definitive checklist of required atomic visual primitives,such as key objects and activities, along with their temporal ordering.These systematically derived grounding events provide the critical feed-back signal for RL post-training, enabling dense, verifiable rewards basedon the successful retrieval of these specific visual elements rather thanrelying entirely on outcome-only answer accuracy. By explicitly optimiz-ing for this verifiable evidence-seeking behavior, VSeek improves Pass@1scores by up to 8% and Pass@4 scores by 15% on long-video understand-ing benchmarks compared to base models. We open-source our code athttps://utaustin-swarmlab.github.io/VSeek.