G3AFT: Glance Guided Gradient Aligned Fine-Tuning for Visual Autoregressive Models

Autoregressive models in vision have gained widespread at-tention under the scalable next-token prediction paradigm, representingimages as sequences of discrete tokens. Yet their optimization remainschallenging due to the non-differentiability of discrete token samplingand the prohibitive memory costs of full-sequence fine-tuning. We pro-pose G3 AFT, a novel end-to-end fine-tuning framework for discrete-tokenautoregressive models. Within this framework, we introduce two com-plementary techniques, i.e. Global Scope Gradient Recording (GSGR)and Noise Suppressed Gradient Projection (NSGP). GSGR selectivelyupdates periodically sampled tokens by leveraging coarse global signalsthat summarize the overall layout and long-range structure of the image.This design alleviates the prohibitive memory overhead of full-sequencefine-tuning while maintaining long-range coherence. In parallel, NSGPenhances surrogate gradient flow by explicitly aligning updates withsalient sampling alternatives and filtering out spurious signals introducedby softmax relaxations. By suppressing noisy gradient components andreinforcing causally consistent updates, NSGP provides a more faithfulapproximation to discrete sampling, ensuring faithful gradient propaga-tion across long autoregressive trajectories. To the best of our knowl-edge, this is the first method under the next-token prediction paradigmfor discrete-token autoregressive image generation that enables fully dif-ferentiable optimization across the entire generation process. Extensiveexperiments validate that G3 AFT achieves superior image quality andquantitative performance, providing a practical framework for fully dif-ferentiable fine-tuning of autoregressive image generation models. Codeis available at https://github.com/Wotoosh/G3AFT.