D2PO: Optimizing Diffusion Samplers via Dynamic Preference

We propose D2PO (Dynamic Direct Preference Optimiza-tion), a principled framework for optimizing diffusion sampling policieswith respect to timestep schedules and classifier-free guidance (CFG)weights. Our work is motivated by a fundamental limitation of existingstudent-teacher regression frameworks; low-NFE student samplers aretrained to mimic high-NFE teachers, often sacrificing high-frequency tex-ture fidelity while preserving coarse global structures, thereby misalign-ing the sampler with perceptual quality. D2PO addresses this challengeby reformulating sampler optimization as a preference-based alignmentproblem, leveraging the Direct Preference Optimization (DPO) frame-work. To make DPO applicable to diffusion samplers, we model thesampling policy as an energy-based model (EBM), transforming pref-erence comparisons into tractable energy differences. We further intro-duce a novel energy formulation derived directly from the pretrainedscore network, enabling preference evaluation in perturbed spaces thatjointly capture structural consistency and fine-grained details. More-over, we introduce dynamic preferences, where the preferred samplesused for alignment progressively improve as the sampling policies arelearned. This self-improving mechanism replaces rigid static teacher su-pervision with an iterative, preference-guided refinement process, pro-viding progressively stronger alignment signals. Extensive experimentsdemonstrate that D2PO aligns diffusion samplers with perceptual qual-ity more faithfully, unlocking the full potential of high-quality teachersand consistently outperforming conventional regression-based schedulersunder low-NFE constraints.