SpecEyes: Accelerating Agentic Multimodal LLM via Speculative Planning and Perception

Agentic multimodal large language models (MLLMs) (e.g.,OpenAI o3 [41] and Gemini Agentic Vision [10]) achieve remarkablereasoning capabilities through the iterative invocation of visual tools.However, the cascaded perception, reasoning, and tool-calling loops intro-duce significant sequential overhead. This overhead, termed agentic depth,incurs prohibitive latency and seriously limits system-level concurrency.To this end, we propose SpecEyes, an agentic-level speculative accelera-tion framework that breaks this sequential bottleneck. Our key insight isthat a lightweight MLLM can plan a tool-free execution path to directlyanswer many queries, bypassing the expensive tool-use loop. To regulatethis speculative planning, we introduce a cognitive gating mechanismbased on answer separability, which quantifies the model’s confidence inself-verification without requiring oracle labels. Furthermore, we designa heterogeneous parallel funnel that exploits the small model’s statelessconcurrency to mask the large model’s stateful serial execution, therebymaximizing system throughput. Extensive experiments on V* Bench,HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1 − 3.35×speedup over the baseline while preserving or even improving accuracy,thereby boosting serving throughput under concurrent workloads.