Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

Vision–Language Models (VLMs) excel at reasoning in lin-guistic space but struggle with perceptual understanding that requiresdense visual perception, e.g., spatial reasoning and geometric aware-ness. This limitation stems from the fact that current VLMs have lim-ited mechanisms to capture dense visual information across spatial di-mensions. We introduce Chain-of-Visual-Thought (CoVT), a frameworkthat enables VLMs to reason not only with discrete text tokens but alsothrough continuous visual tokens—compact latent representations thatencode rich perceptual cues. With a small budget of roughly 20 tokens,CoVT distills knowledge from lightweight vision experts that capturecomplementary properties such as 2D appearance, 3D geometry, spatiallayout, and edge structure. During training, the VLM with CoVT au-toregressively predicts these visual tokens to reconstruct dense supervi-sion signals (e.g., depth, segmentation, edges, and DINO features). At in-ference, the model reasons directly in the continuous visual latent space,preserving efficiency while optionally decoding dense predictions for in-terpretability. Evaluated across more than ten diverse benchmarks, in-cluding CV-Bench, MME-RealWorld, MMVP, RealWorldQA, MMStar,WorldMedQA, and HRBench, integrating CoVT into strong VLMs suchas Qwen2.5-VL and LLaVA consistently improves performance by 3% to16% and demonstrates that compact continuous visual thinking enablesmore precise, grounded, and interpretable multimodal intelligence.