Task Alignment: A simple and effective proxy for model merging in computer vision

Efficiently merging several models fine-tuned for differenttasks, but stemming from the same pretrained base model, is of greatpractical interest. Despite extensive prior work, most evaluations of modelmerging in computer vision are restricted to image classification usingCLIP, where different classification datasets define different tasks. In thiswork, our goal is to make model merging more practical and show itsrelevance on challenging scenarios beyond this specific setting. In mostvision scenarios, different tasks rely on trainable and usually heteroge-neous decoders. Differently from previous studies with frozen decoders,where merged models can be evaluated right away, the non-trivial cost ofdecoder training renders hyperparameter selection based on downstreamperformance impractical. To address this, we introduce the task align-ment proxy, and show how it can be used to speed up hyperparameterselection by orders of magnitude while retaining performance. Equippedwith the task alignment proxy, we extend the applicability of modelmerging to multi-task vision models beyond CLIP-based classification.