DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space

Existing latent diffusion models excel at visual generationand editing tasks, employing autoencoders to project RGB images andvideos into latent spaces. However, their low-compression latents pro-duce redundant and excessive visual tokens, limiting scalability to high-resolution content such as 4K images and 2160p videos. One promis-ing approach for acceleration is to leverage deeply compressed latentspaces produced by deep compression autoencoders. However, trainingindustrial-scale models from scratch remains expensive and often fails tomatch the performance of pre-trained models. Alternatively, directly fine-tuning existing pre-trained models on deeply compressed latent spacesis appealing, but it introduces a critical challenge: a representation gapbetween the original latent space and the target compressed space, whichcan result in suboptimal convergence and even training instability. DC-Gen offers the first solution to tackle this challenge. Specifically, we in-troduce a novel embedding alignment process to bridge the representa-tion gap. Once the embedding spaces are aligned, only lightweight LoRAfine-tuning is required to unlock the pre-trained model’s full generativecapability. We validate the effectiveness of DC-Gen across tasks rangingfrom image and video generation to image editing. The resulting DC-Genmodels achieve quality comparable to pre-trained models while deliveringsubstantial speedups. For example, DC-Gen-FLUX reduces the latencyof 4K image generation by 53.8× on a single NVIDIA H100 GPU. Codeis available at https://github.com/dc-ai-projects/DC-Gen.