Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale

Differentially private training adds isotropic Gaussian noiseto clipped gradients, corrupting every singular direction equally. In vi-sion models, where spatial correlation concentrates gradient energy intoa low-rank subspace, most of this noise falls in directions that carrylittle signal. Spectral gradient orthogonalization via polar decomposi-tion is introduced as a post-processing step that recovers directionalsignal from the noisy gradient’s low-rank structure at zero additionalprivacy cost. A phase transition governs the utility of this approach:orthogonalization improves accuracy only when the per-direction spec-tral signal-to-noise ratio (SNR) suffices for singular vector recovery; inlow-SNR regimes, the directional bias of the gradient is replaced by anearly random orthogonal update, and the transformation is harmful.The recovery threshold is determined by the spectral gap of the gradientand is surpassed at large batch sizes. Empirically, the benefit scales withmodel capacity: spectral orthogonalization achieves a +20.9% improve-ment over DP-SGD on WRN-28-10 (B = 4096) and +14.9% on ResNet-18, while reducing inter-run variance by a factor of two to three. In thefine-tuning regime, spectral orthogonalization matches the stability ofDP-Adam while maintaining a first-order memory footprint. Combiningspectral with temporal denoising yields 50.3% on CIFAR-10 (ε = 4), thehighest accuracy in any tested configuration. These gains are specific tomoderate-to-high-SNR regimes such as large-batch training of higher-capacity models. Small-batch or low-SNR settings are better served byDP-SGD or temporal denoising.