Video-Holmes: Can MLLM Think like Holmes for Complex Video Reasoning?

Recent advances in Chain-of-Thought reasoning have beenreported to enhance video reasoning capabilities of multimodal large lan-guage models (MLLMs). This progress naturally raises a question: Canthese models perform complex video reasoning in a manner comparable tohuman experts? However, existing video benchmarks primarily evaluatevisual perception and grounding, with questions that can be answeredbased on explicit prompts or isolated visual cues (e.g., “What is thewoman wearing?”). Such benchmarks do not fully capture the intricaciesof real-world reasoning, where humans must actively search for, integrate,and analyze multiple clues before reaching a conclusion. To address this,we introduce Video-Holmes, a benchmark inspired by the reasoningprocess of Sherlock Holmes, designed to evaluate the complex video rea-soning capabilities of MLLMs. Video-Holmes consists of 1,837 questionsderived from 270 manually annotated suspense short films, spanningseven carefully designed tasks. Each task is constructed by first identify-ing key events and causal relationships within films, and then designingquestions that require models to actively locate and connect multiplerelevant visual clues scattered across video clips. Evaluation of state-of-the-art MLLMs reveals that while these models generally excel at vi-sual perception, they encounter difficulties with integrating informationand often miss critical clues. For example, the best-performing model,Gemini-3.0-Pro, achieves an accuracy of only 49.6%, with most modelsscoring below 40%. We aim for Video-Holmes to serve as a “Holmes-test”for multimodal reasoning, motivating models to reason more like humansand emphasizing the ongoing challenges in this field.