Multi-dimensional Preference Alignment by Conditioning Reward Itself

Reinforcement Learning from Human Feedback has emergedas a standard for aligning di!usion models. However, we identify a fun-damental limitation of the standard DPO formulation when optimizingmulti-dimensional preferences: by relying on the Bradley-Terry model,it aggregates heterogeneous evaluation axes (e.g., aesthetic quality andsemantic alignment) into a single scalar reward. This aggregation createsa reward conflict where the model is forced to unlearn desirable featuresof a specific dimension if they appear in a globally non-preferred sam-ple. To address this issue, we propose Multi Reward Conditional DPO(MCDPO) which resolves reward conflicts by introducing a disentangledBradley-Terry objective. MCDPO explicitly injects a preference outcomevector as a condition during training, which allows the model to learnthe correct optimization direction for each reward axis independentlywithin a single network. Extensive experiments on Stable Di!usion 1.5and SDXL demonstrate that MCDPO achieves superior performance onbenchmarks even with 18% and 3% training data respectively. Notably,our conditional framework enables dynamic and multiple-axis control atinference time using Classifier Free Guidance to amplify specific rewarddimensions without additional training or external reward models.