TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning

The frontier of visual reasoning is shifting toward models likeOpenAI o3, which can intelligently create and operate tools to trans-form images for problem-solving, also known as thinking-with-images inchain-of-thought. Yet existing benchmarks fail to fully capture this ad-vanced capability. Even Visual Search, the most common benchmark forcurrent thinking-with-images methods, tests only basic operations suchas localization and cropping, offering little insight into more complex,dynamic, and tool-dependent reasoning. We introduce TIR-Bench, acomprehensive benchmark for evaluating agentic thinking-with-imagesacross 13 diverse tasks, each requiring novel tool use for image process-ing and manipulation in chain-of-thought. We evaluate 22 multimodallarge language models (MLLMs), from leading open-sourced and propri-etary models to those with explicit tool-use augmentation. Results showthat TIR-Bench is universally challenging, and strong performance re-quires genuine thinking-with-images capabilities. Finally, we present apilot study comparing direct versus agentic fine-tuning.