Co-evolving Representations in Joint Image-Feature Diffusion

Joint image–feature generative modeling has recently emergedas an effective strategy for improving diffusion training by coupling low-level VAE latents with high-level semantic features extracted from pre-trained visual encoders. However, existing approaches rely on a fixedrepresentation space, constructed independently of the generative objec-tive and kept unchanged during training. We argue that the represen-tation space guiding diffusion should itself adapt to the generative task.To this end, we propose Coevolving Representation Diffusion (CoReDi),a framework in which the semantic representation space evolves duringtraining by learning a lightweight linear projection jointly with the diffu-sion model. While naïvely optimizing this projection leads to degeneratesolutions, we show that stable coevolution can be achieved through acombination of stop-gradient targets, normalization, and targeted reg-ularization that prevents feature collapse. This formulation enables thesemantic space to progressively specialize to the needs of image synthesis,improving its complementarity with image latents. We apply CoReDi toboth VAE latent diffusion and pixel-space diffusion, demonstrating thatadaptive semantic representations improve generative modeling acrossboth settings. Experiments show that CoReDi achieves faster convergenceand higher sample quality compared to joint diffusion models operatingin fixed representation spaces.