ARVAR: Accelerating Visual Autoregressive Model via Attention Retrospect

Visual autoregressive (VAR) models synthesize images in acoarse-to-fine manner by predicting multi-scale token maps, achievingcompetitive text-to-image quality with fewer decoding iterations thanraster-scan autoregression. However, late-scale refinement dominates in-ference time because computation grows quadratically with resolutionand attention becomes increasingly costly. Prior accelerations either fil-ter tokens heuristically, risking severe artifacts, or rely on extra deci-sion models that require per-model fine-tuning with reduced flexibility.We present the first in-depth study of attention behavior across VARgeneration steps and uncover two key properties: (1) attention patternscorresponding layers remain consistent across steps; (2) conditional andunconditional branches progressively converge at later scales. Leveragingthese insights, we introduce a novel training-free, flashattention-friendlyacceleration recipe ARVAR with two primary components: (1) Atten-tion Retrospect (AR) trims tokens at high resolution phase by to-ken saliency from early stage; (2) Branch Halving (BH) drops oneof the conditional/unconditional branches in the late steps. Combined,AR and BH deliver faster and higher-quality autoregressive image gen-eration, achieving 1.56× and 3.01× speedups on HART and Infinityrespectively with negligible performance drop. Compared with existingacceleration methods, ARVAR substantially improves image fidelity.