GR-GRPO: Graph-Diffused Credit for Autoregressive Image RL Alignment

Reinforcement-learning alignment for autoregressive imagegeneration is bottlenecked by expensive rollouts: in GRPO-style groupoptimization, sampling G rollouts per prompt improves exploration andwithin-group credit estimation but multiplies sampling cost. This chal-lenge is amplified by vector-quantized (VQ) tokenization, where nearbycodebook IDs in embedding space can be locally substitutable; however,GRPO-style updates provide explicit positive credit only to sampled IDs,leading to insufficient coverage of success-supported alternatives undersmall-G training. We propose GR-GRPO (Graph-Regularized GRPO),which densifies token-level supervision without additional rollouts by dif-fusing positive evidence from positive rollouts over a precomputed code-book K-NN graph to construct graph-diffused soft targets, and regulariz-ing the policy toward these targets via an auxiliary cross-entropy term. Aconfidence-adaptive gate modulates diffusion strength based on the pol-icy’s logit margin, and a difficulty-aware budget allocation scheme fur-ther improves rollout utilization across prompts. Extensive experimentson compositional image generation (GenEval) and preference-style align-ment (DrawBench) validate the effectiveness of GR-GRPO. In particular,GR-GRPO achieves a substantially better compute–performance trade-off, surpassing large-group training (G=64) with ∼ 4.6× less rollout timeon Janus-Pro-1B. Our code and model weights are available athttps://github.com/vivoCameraResearch/GR-GRPO.