PhenoLIP: Phenotype Guided Medical Vision–Language Pretraining

Recent progress in CLIP-like vision-language models (VLMs)has greatly advanced medical image analysis. However, most existingmedical VLMs still rely on coarse image-text contrastive objectives andfail to capture the systematic visual knowledge encoded in well-definedmedical phenotype ontologies. To address this limitation, we constructPhenoKG, the first large-scale, phenotype-centric multimodal knowl-edge graph that encompasses around 524K high-quality image-text pairslinked to more than 3,000 phenotypes. Building upon PhenoKG, we pro-pose PhenoLIP, a pretraining framework that explicitly incorporatesstructured phenotype knowledge into medical VLMs through a two-stage process. We first learn a knowledge-enhanced phenotype embed-ding space that captures the hierarchical structure of phenotype ontolo-gies and then distill this textual knowledge into multimodal pretrainingvia a teacher-guided knowledge distillation objective. To support eval-uation, we further introduce PhenoBench, an expert-verified bench-mark designed for phenotype recognition, comprising over 7,800 image-caption pairs covering more than 1,000 phenotypes. Extensive experi-ments demonstrate that PhenoLIP outperforms previous state-of-the-artbaselines, improving upon BIOMEDICA in phenotype classification ac-curacy by 8.06% and in cross-modal retrieval by 15.03%, underscoringthe value of integrating phenotype-centric priors into medical VLMs forstructured and interpretable medical image understanding.