DRPO: Disentangling Demographic Bias from Rewards for Fair Diffusion Alignment

Text-to-image diffusion models are increasingly aligned with human preferences through preference optimization (PO) with learned reward models. We observe that the CLIP-based reward models used in this pipeline encode demographic biases, and that PO amplifies them: after alignment, the demographic distribution of generated images narrows significantly compared to the base model. We show that this degradation is progressive—race entropy drops steadily throughout training— and that existing diversity-preservation methods such as DRIFT and DiverseGRPO do not resolve it, because visual diversity and demographic fairness are distinct properties. To address this, we propose DRPO, which identifies demographic directions in the reward feature space via linear probing and removes them through orthogonal projection before reward computation. DRPO requires no demographic annotations during training and is compatible with any PO method. Experiments on Stable Diffusion and SDXL with six baselines demonstrate that DRPO maintains generation quality while substantially restoring the demographic balance lost during alignment.