ORFC: Orthogonal Reparameterization for Low-Bitrate ViT Feature Coding

Existing feature coding methods typically minimize feature-space reconstruction error, implicitly assuming that preserving featuresimilarity ensures downstream task performance. However, in VisionTransformers (ViTs), global correlation mixing makes downstream pre-dictions more sensitive to small feature perturbations, so similar fea-ture distortion can lead to significantly different task performance. Inparticular, our analysis reveals that under fixed-group product quanti-zation, task sensitivity is highly uneven across channel subspaces. Con-sequently, near-uniform per-group rates may waste bits on insensitivedirections while leaving critical ones underrepresented, causing rapidtask-performance degradation at low bitrate. To address this issue, wepropose an Orthogonal Reparameterized product quantization for inter-mediate Feature Coding (ORFC). It introduces a learnable orthogonaltransform that rotates the representation before grouped quantization,enabling fixed-size groups to better align with task-relevant directions.We further use the output deviation of the remaining layers as a proxydistortion, and optimize an entropy-constrained rate-distortion objectiveto jointly learn the orthogonal transform and grouped quantization in anend-to-end manner. Extensive experiments across multiple ViT architec-tures and downstream tasks demonstrate that ORFC achieves a superiorrate-task trade-off, especially in the low-bitrate regime. The code is avail-able at https://github.com/zhangletian2/ORFC.