Towards Reliable Medical Large Vision-Language Models via Counterfactual Preference Optimization

Medical Large Vision-Language Models (Med-LVLMs) haveemerged as powerful tools for medical image understanding and au-tomated reporting. However, existing models often suffer from modal-ity bias, where correlations between background context and diagnos-tic labels cause over-reliance on spurious priors rather than lesion evi-dence, leading to clinically irrelevant or hallucinated outputs. To addressthis, we propose a Counterfactual Medical Preference Optimization(CoMedPO) framework that learns an unbiased policy from a biasedreference model. Unlike conventional direct preference optimization (DPO)methods that inherit dataset biases, CoMedPO mitigates spurious cor-relations while preserving the valuable indirect effect between lesionand background dependencies in medical images. This causal formula-tion yields a new preference-based loss with theoretical guarantees forbias mitigation and robust policy learning. Our framework is model-agnostic and seamlessly integrates with existing Med-LVLM alignmentpipelines. Extensive experiments on Med-VQA and medical report gen-eration demonstrate that CoMedPO consistently improves factual accu-racy and clinical relevance, outperforming state-of-the-art DPO variants.These results suggest that CoMedPO provides a principled route towardtrustworthy and clinically grounded multimodal medical reasoning. Ourcode is available at https://github.com/zxgapollo/CoMedPO.