From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

Although Multimodal Large Language Models (MLLMs) havedemonstrated impressive performance in Medical Visual Question An-swering (Med-VQA), their reliance on global image features often lacksprecise pixel-level grounding, thereby limiting clinical trustworthiness.To bridge the semantic gap between high-level clinical reasoning andspatial localization, we propose MedREAL (Medical REasoning-drivenAnswering and Localization), a unified framework that seamlessly alignslinguistic reasoning with spatial grounding. Specifically, MedREAL in-troduces Seg Anchored Reasoning Pooling (SARP) to distill task-relevantsemantic evidence directly from [SEG] tokens within the MLLM’s hid-den states. Furthermore, a Reasoning-to-Visual (R2V) fusion mecha-nism is proposed to effectively inject these reasoning-aware features intoa segmentation pipeline for accurate mask decoding. To facilitate thisparadigm, we construct MedRAVS-13K, a comprehensive dataset com-prising 13,824 expertly validated samples across four diverse imagingmodalities. Extensive experiments demonstrate that MedREAL signifi-cantly outperforms state-of-the-arts, achieving 68.49% gIoU and 70.47%cIoU on benchmark evaluations. By generating evidence masks that arestrictly consistent with textual diagnoses, MedREAL provides a robust,interpretable framework for reasoning-driven medical image analysis.