SRRA: Stable-Rank-Based Residual Adaptation for Generalizable Deepfake Detection

Deepfake detection faces challenges in generalization, prompt-ing recent studies to fine-tune pretrained Vision Transformers for im-proved performance. Existing fine-tuning-based detection approaches of-ten assign the same rank to the tunable residual components of alllayers. However, the shallow layers exhibit a distinctly low-rank struc-ture, whereas the middle and deep layers present more complex struc-tures. Therefore, we argue that only slight fine-tuning is required forthe shallow layers, which are able to effectively capture low-level forgerycues, while the middle and deep layers require higher residual ranks tomodel long-range dependencies and extract high-level forgery artifacts.To achieve these goals adaptively, we propose the Stable-Rank-BasedResidual Adaptation (SRRA) strategy. It utilizes singular value decom-position to decompose the pretrained weight matrices and adaptively al-locates frozen principal components and learnable residual componentsbased on the Stable Rank. In addition, we design the Residual En-ergy Constraint (REC) and Residual Subspace Orthogonalization (RSO),which suppress excessive perturbations and promote subspace decou-pling, avoiding overfitting and improving the generalization performance.Extensive experiments demonstrate that SRRA achieves remarkable im-provements in cross-domain generalization across multiple deepfake de-tection benchmarks. Our code is available at https://github.com/LHK-CodeLab/SRRA.