ELT: Elastic Looped Transformers for Visual Generation

We introduce Elastic Looped Transformers (ELT), a highlyparameter-efficient class of visual generative models based on a recurrenttransformer architecture. While conventional generative models rely ondeep stacks of unique transformer layers, our approach employs iterative,weight-shared transformer blocks to drastically reduce parameter countswhile maintaining high synthesis quality. To effectively train these modelsfor image and video generation, we propose the idea of Intra-Loop SelfDistillation (ILSD), where student configurations (intermediate loops)are distilled from the teacher configuration (maximum training loops)to ensure consistency across the model’s depth in a single training step.Our framework yields a family of elastic models from a single trainingrun, enabling Any-Time inference capability with dynamic trade-offs be-tween computational cost and generation quality, with the same parame-ter count. ELT significantly shifts the efficiency frontier for visual synthe-sis. With 4 × reduction in parameter count under iso-inference-computesettings, ELT achieves a competitive FID of 2.0 on class-conditional Im-ageNet 256 × 256 and FVD of 72.8 on class-conditional UCF-101.Vanilla Looped Transformers Elastic Looped TransformersXk 0 Xk 0ILSDTrainingXk intXk max Xk int Xk maxSolution Space Solution Space✗ Xk int away from solution space. ✓ Xk int moves towards solution space.✗ Fixed Depth Inference ✓ Any-Time Inference.Fig. 2: Latent Trajectories of Standard vs. Elastic Looped Transformers.Xk int & Xk max represent output of intermediate (Lint ) & final loops (Lmax ) respectivelyfor kth generation sampling step. Unlike standard recurrent models (left) where onlythe final iteration Xk max reaches the solution space, our ILSD training (right) guidesintermediate states Xk int also toward the target space. This transformation shifts themodel from a fixed-depth architecture to an Any-Time inference framework, supportingflexible computational budgets through early exits within a sampling step.