Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labels

Prompt learning has gained significant attention as a parameter-efficient approach for adapting large pre-trained vision-language modelsto downstream tasks. However, when only partial labels are available, itsperformance is often limited by label ambiguity and insufficient supervi-sory information. To address this issue, we propose Holistic Optimal La-bel Selection (HopS), leveraging the generalization ability of pre-trainedfeature encoders through two complementary strategies. First, we designa local density-based filter that selects the top frequent labels from thenearest neighbors’ candidate sets and uses the softmax scores to identifythe most plausible label, capturing structural regularities in the featurespace. Second, we introduce a global selection objective based on optimaltransport that maps the uniform sampling distribution to the candidatelabel distributions across a batch. By minimizing the expected trans-port cost, it can determine the most likely label assignments. These twostrategies work together to provide robust label selection from both lo-cal and global perspectives. Extensive experiments on eight benchmarkdatasets show that HopS consistently improves performance under par-tial supervision and outperforms all baselines. Those results highlightthe merit of holistic label selection and offer a practical solution forprompt learning in weakly supervised settings. The code is available athttps://github.com/Qizhoay/HopS.