Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling

Visual Autoregressive Models (VAR) generate images throughnext-scale prediction, producing all tokens within each scale in paral-lel. We show that this parallel decoding constitutes a mean-field-styleapproximation that discards spatial dependencies among same-scale to-kens, causing locally incoherent samples regardless of backbone capacity– a limitation of the decoding rule. Addressing this limitation, we intro-duce the Logit Refiner, a lightweight autoregressive module that restoresintra-scale dependencies by sequentially sampling tokens conditioned onfrozen backbone features. Adding only ∼10% parameters and less than5% of the base model’s training compute, it plugs into any pretrainedVAR checkpoint without retraining. Controlled ablations isolate jointintra-scale sampling – rather than additional capacity or training – asthe critical ingredient. Across backbones from 310M to 2B parameterson class-conditional ImageNet 256×256, the refiner consistently improvesgeneration quality, enabling a 1.1B-parameter model to surpass one twiceits size. The approach further generalizes to text-to-image generation,confirming that the mean-field bottleneck persists across VAR variantsand is effectively alleviated by our method.Project page: https://compvis.github.io/logit-refiner/.