RUTaL: Residual Upcycling with Task Ladder for Efficient Multi-Task Learning

Pretrained vision models are widely used as backbones fordownstream tasks, yet adapting them to multi-task learning (MTL) re-mains challenging. Naive full fine-tuning incurs substantial overhead andoften exacerbates task interference, while existing parameter-efficientfine-tuning (PEFT) methods are largely designed for single-task adap-tation. We propose Residual Upcycling with Task Ladder (RUTaL), aparameter-efficient multi-task vision adaptation framework. To accom-modate the diverse and heterogeneous representation demands in MTL,We proposed Task-General Residual Upcycling (TGRU) to transforma pre-trained vision Transformer backbone into a Mixture-of-Expertsarchitecture through low-rank residual weight reparameterization, en-abling efficient and task-scalable capacity expansion. Built upon this up-cycled representation paradigm, we further introduce Task-Specific Lad-der Adaptation (TSLA), which extracts task-relevant features from theshared upcycled representations in a decoupled manner to accommodatethe unique requirements of each task. Experiments on multi-task densescene understanding benchmarks show that RUTaL achieves state-of-the-art performance while demonstrating superior computational efficiency.