Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models

Text-to-image diffusion models deliver high-quality images,yet aligning them with human preferences remains challenging. We re-visit diffusion-based Direct Preference Optimization (DPO) for thesemodels and identify a critical pathology: enlarging the preference mar-gin does not necessarily improve generation quality. In particular, thestandard Diffusion-DPO objective can increase the reconstruction er-ror of both winner and loser branches. Consequently, degradation ofthe less-preferred outputs can become sufficiently severe that the pre-ferred branch is also adversely affected even as the margin grows. Toaddress this, we introduce Diffusion-SDPO, a safeguarded update rulethat preserves the winner by adaptively scaling the loser gradient ac-cording to its alignment with the winner gradient. A first-order analysisyields a closed-form scaling coefficient that guarantees the error of thepreferred output is non-increasing at each optimization step. Our methodis simple, model-agnostic, broadly compatible with existing DPO-stylealignment frameworks and adds only marginal computational overhead.Across standard text-to-image benchmarks, including validations on thelarge-scale FLUX.1-dev model, Diffusion-SDPO delivers consistent gainsover preference-learning baselines on automated preference, aesthetic,and prompt alignment metrics. Our code is available at https://github.com/AIDC-AI/Diffusion-SDPO.