GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models

The text encoder is a critical component of text-to-image andtext-to-video diffusion models, fundamentally determining the semanticfidelity of the generated content. However, its development has been hin-dered by two major challenges: the lack of an efficient evaluation frame-work that reliably predicts downstream generation performance, and thedifficulty of effectively adapting pretrained language models for visualsynthesis. To address these issues, we introduce GRAN-TED, a paradigmto Generate Robust, Aligned, and Nuanced Text Embeddings for Diffu-sion models. Our contribution is twofold. First, we propose TED-6K, anovel text-only benchmark that enables efficient and robust assessmentof an encoder’s representational quality without requiring costly end-to-end model training. We demonstrate that performance on TED-6K,standardized via a lightweight, unified adapter, strongly correlates withan encoder’s effectiveness in downstream generation tasks. Notably, un-der our experimental setup, compared with training a diffusion modelfrom scratch, evaluating with TED-6K is about 750× faster. Second,guided by this validated framework, we develop a superior text encoderusing a novel two-stage training paradigm. This process involves an ini-tial fine-tuning stage on a Multimodal Large Language Model for bettervisual representation, followed by a layer-wise weighting method to ex-tract more nuanced and potent text features. Our experiments show thatthe resulting GRAN-TED encoder not only achieves state-of-the-art per-formance on TED-6K but also leads to demonstrable performance gainsin text-to-image and text-to-video generation.