D-VLAM: Differential Vision and Language Mixing for Rehearsal Free Continual Learning

Language-guided prompt-based continual learning emergesas a promising approach to tackle catastrophic forgetting in a dynamicenvironment under rehearsal-free constraint. However, existing state-of-the-art (SOTA) requires additional resources, such as LLM-generateddescriptors and an LLM decoder that may not be available in a real ap-plication. Meanwhile, the more straightforward method experiences sub-optimality due to the high similarity between language text embeddings.To address this problem, we propose a resource-efficient novel language-guided approach and algorithm that incorporates two novel ideas: (1)mixed vision and language modality for prompt generation, and (2) Dif-ferential prefix tuning for the model training process. Our experimentalanalysis shows that our method outperforms existing language-guidedprompt-based methods, i.e., up to 30%, 26%, and 10% for final aver-age accuracy, cumulative average accuracy, and final forgetting mea-sure, respectively. The historical analysis confirms our method’s stability-plasticity balance in every task. Our extended analysis shows that ourmethod consistently achieves better performance in various prompt lengthsand ViT layers. For further study and reproducibility, we also providerigorous analysis, details, and source code of our method in the sup-plementary document. The implementation of the proposed method isavailable at https://github.com/anwarmaxsum/D-VLAM.