To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models

Test-time adaptation (TTA) has emerged as a prominentstrategy for adapting vision-language models to distribution shifts dur-ing inference. We conduct a per-sample analysis of model predictionsbefore and after adaptation, and observe two failure modes in existingTTA methods that echo previous work. Adaptations are frequently neg-ligible, yielding no change in the model’s predictions, and more severely,they can be detrimental by flipping previously correct predictions to in-correct ones. This naturally raises a question: Can we identify and skipsuch negligible or harmful adaptations? In this work, we introduce a newproblem of selective adaptation, which aims to determine whether agiven test sample should undergo adaptation or be skipped. To this end,we propose Cross-Augmentation Similarity (CAS), a simple baseline thatperforms adaptation only when predictions across augmented views ex-hibit low similarity. Notably, CAS not only preserves but in some casesimproves overall accuracy, even when skipping nearly 85% of the adap-tation process. We hope other researchers will explore this new directionand surpass the performance of our baseline. Our code is available athttps://github.com/sirujiang/selective-adaptation.