DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

Few-shot Test-Time Domain Adaptation (FSTT-DA) seeksto adapt models to novel domains using only a handful of unlabeledtarget samples. This setting is more realistic than typical domain adap-tation setups, which assume access to target data during source training.However, prior FSTT-DA approaches fail to effectively leverage sourcedomain-specific knowledge, relying on shallow batch normalization up-dates, prompt-based methods that treat the model as a black box, orensembling strategies that do not capture cross-domain relationships. Toaddress these limitations, we introduce a new FSTT-DA framework thatintegrates LoRA fine-tuning with model merging. In our approach, sep-arate LoRA modules are fine-tuned on CLIP’s vision encoder for eachsource domain. Since LoRA modifies only a small fraction of the model’sparameters, it retains the base model’s generalized knowledge while in-ternally learning domain-specific features. To adapt the learned knowl-edge to a specific target domain, we propose a hypernetwork trainedvia meta-learning that generates per-column merging factors to combineLoRA modules. Given a small batch of target images, the hypernetworkproduces merging weights that fuse source LoRA modules into a singleadapted representation. Our results demonstrate state-of-the-art perfor-mance across various domain adaptation datasets. Our code is publiclyavailable at https://github.com/nahbois4321/DA-MergeLoRA.