The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Vision-language models (VLMs) are increasingly deployedin high-stakes settings where reliable uncertainty quantification (UQ)is as important as predictive accuracy. Extended reasoning via chain-of-thought (CoT) prompting or reasoning-trained models has becomeubiquitous in modern VLM pipelines, yet its effect on UQ reliabilityremains poorly understood. Our results show that reasoning tends todegrade the quality of many uncertainty estimates, even when it im-proves task accuracy. We identify implicit answer conditioning as theprimary mechanism: as reasoning traces converge on a conclusion beforethe final answer is generated, token probabilities increasingly reflect con-sistency with the model’s own reasoning trace rather than uncertaintyabout correctness. In effect, the model becomes overconfident in its an-swer. In contrast, agreement-based consistency remains robust and oftenimproves under reasoning, making it a practical choice for uncertaintyestimation in reasoning-enabled VLMs.