CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

Chain-of-thought (CoT) reasoning has enabled multi-modallarge language models (MLLMs) to tackle complex visual reasoning tasksby generating explicit intermediate reasoning steps in natural language.However, this text-based reasoning paradigm is inherently slow at infer-ence time with even thousands of tokens and fundamentally constrainedby the expressiveness of natural language. In this paper, we proposeCoLT (Chain of Latent Thoughts), a novel framework that teachesmulti-modal models to reason through a chain of latent thought repre-sentations instead of verbose text tokens, which can perform thinkingwith as few as 3 steps. Naively forcing the model to think with latentstates easily produces meaningless semantics and makes training unsta-ble. To effectively regulate the latent reasoning process, we introduce alightweight external decoder that provides step-level supervision for eachlatent reasoning step in two complementary directions: a forward modethat decodes latent thoughts into the textual reasoning of the next step,and a backward mode that aligns decoder hidden states with the model’slatent thoughts given preceding textual context. We further incorporateinternal supervision that encourages coherent step-by-step latent transi-tions. The decoder and internal supervision are removed during inferenceto maintain high efficiency of latent reasoning. Extensive experiments oneight benchmarks demonstrate that CoLT not only outperforms existinglatent reasoning methods such as CODI and SIM-CoT, but also surpasseslatent visual reasoning approaches that rely on auxiliary images withcostly annotation requirements. Compared to text CoT methods, CoLTcan notably reduce the inference time by 10.1× and text decoding timeby 22.6×. Code is released at https://github.com/hulianyuyy/CoLT.