Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis

Diffusion models have achieved remarkable success across di-verse domains, with performance closely related to the denoising back-bones that parameterize the score function. In this paper, we presenta systematic, phase-aware analysis of diffusion components and showthat abrupt, early-stage fluctuations in deep latents are strongly asso-ciated with artifacts. Guided by these findings, we introduce DUNE(Diffusion Unified Network refiNEr), a training-free refinement frame-work that detects abrupt deviations in deep low-noise internal latentsusing a shared EMA-based criterion, and applies backbone-specific sup-pression to the detector-selected entries. Although derived from U-Net,the same detect–suppress principle extends naturally to Transformer-based diffusion models by acting on the latents of deep self-attentionblocks. Extensive experiments across multiple backbones indicate thatDUNE improves fidelity while reducing hallucinations, offering new in-sight into where and when diffusion backbones should be controlled.