RotateAttention : RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

In DiT-based video generation models equipped with3D Rotary Position Embeddings (3D RoPE), the attention mech-anism remains a primary computational bottleneck due to its quadraticcomplexity with respect to sequence length. While quantized FlashAt-tention ox001Bers a promising path toward hardware acceleration, exist-ing low-bit quantization methods overlook two critical challenges in thissetting: 1) applying online rotation matricesx0016a widely used techniquefor mitigating outliers in Queries (Q) and Keys (K )x0016is dix001Ecult toreconcile with RoPE; and 2) the non-negative attention matrix P =exp(QK − max(QK)) makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributionsof Q and K are strongly ax001Bected by the dimensional partitioning of 3DRoPE. Based on this x001Cnding, we propose RotateAttention, an ex001E-cient mixed-precision INT4 FlashAttention framework tailored forDiT-based video generation models with 3D RoPE, using selec-tive FP16 fallback for accuracy-sensitive attention blocks and denoisingsteps. RotateAttention introduces two core techniques: 1) RoPE-awareRotation, which employs either mergeable rotation matrices that canbe fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in Q and K ; and 2) Range-optimized P Quantiza-tion, which uses x001Cxed scales and zero-points to fully exploit the INT4numerical range with minimal computational overhead. Experimentsshow that RotateAttention preserves video generation quality nearlyidentical to full-precision baselines while achieving up to 1.68× end-to-end speedup and 2.2× kernel-level acceleration.