Self-Improving Diffusion Classifiers with Minority Preference Optimization

Prior studies have demonstrated that diffusion classifiers achieverobust zero-shot classification performance. However, their effectivenessis strongly tied to the pretraining data distribution: they perform well inmajority, high-density regions of the data manifold, but are significantlyless accurate in minority, low-density regions. Although prior works onminority sampling have focused on generating more minority-like im-ages, what minority sampling fundamentally enables beyond generationremains underexplored. In this paper, we reveal a direct relationshipbetween minority sampling in generation and the perception capabil-ity of diffusion classifiers. Specifically, we show that enhancing minoritysampling broadens the coverage of underrepresented regions on the datamanifold, thereby improving diffusion-based recognition. To exploit thisconnection, we propose Self-Improving Diffusion Classifiers with Minor-ity Preference Optimization (MiPO), which fine-tunes a pretrained dif-fusion model using minority preference rewards. Using only arbitrarycaption data, MiPO generates candidate samples, rewards those thatbetter cover minority regions, and optimizes the model with LoRA andGroup Relative Policy Optimization, without additional image data, ex-ternal foundation models, or external reward models. This enables stable,prompt-adaptive minority sampling and translates low-density genera-tive coverage into improved zero-shot diffusion classification. To sum up,we show that diffusion classifier perception is biased toward majorityregions, demonstrate that this bias can be alleviated through minoritypreference optimization, and evaluate MiPO on five standard datasets.