Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation

Despite significant advances in Medical Report Generation(MRG), the reliability remains constrained by the prevalence of factualerrors. While Direct Preference Optimization (DPO) has emerged as apromising post-training paradigm to enhance the performance of Super-vised Fine-Tuned (SFT) MRG models, existing DPO-based MRG meth-ods typically adopt a naive preference construction that directly pairsmodel-generated reports with ground truth reports. This strategy inad-vertently entangles critical clinical findings with clinically irrelevant lin-guistic characteristics, and fundamentally lacks explicit vision-languagealignment. To address these challenges, we propose DPO-Clin, a novelpost-training framework that focuses preference optimization on clinicalfindings and cross-modal alignment. First, we introduce the Entity-levelClinical Diagnostic (ECD) module to perform a precise entity-level fac-tual diagnosis. ECD guides the generation of linguistically-aligned re-port preference pairs, isolating clinical discrepancies from linguistic vari-ations. Second, to achieve fine-grained cross-modal alignment, we developM2 DPO, a retrieval-augmented multi-modal DPO variant that enforcestextual preference inversion triggered by visual context switches. Third,we locate correct yet highly uncertain predicted entities and apply coun-terfactual modifications to construct targeted preference data for latentrisk mitigation, thereby further enhancing the model reliability. Exten-sive experiments on two public chest X-ray datasets (MIMIC-CXR andIU X-Ray) and an in-house endoscopy dataset demonstrate that DPO-Clin significantly improves the SFT baselines on clinical-aware metrics.Furthermore, it achieves superior performance over existing DPO-basedMRG methods, exhibiting robust generalizability across distinct baselinearchitectures and diverse medical imaging modalities.