Multi-Anchor Distillation with Text-Guided Analytic Classifier for Continual Learning

Continual learning suffers from two forms of catastrophic forgetting: encoder drift, where visual representations shift as the model adapts to new tasks, and classifier overwriting, where gradient-based updates distort decision boundaries for previously learned classes. We present a unified exemplar-free framework that tackles both effects at the encoder and classifier levels. For the encoder, multi-anchor distillation maintains consistency with multiple historical encoder states and effectively bounds cumulative feature drift along the task sequence. For the classifier, a text-guided analytic classifier computes weights in closed form from incrementally accumulated statistics, completely bypassing gradient-based optimization. To further enhance the classifier, we incorporate a text prior that anchors classifier weights near CLIP’s text embeddings to inject cross-modal semantic knowledge, and propose text subspace regularization that encourages the classifier to learn directions orthogonal to the text embedding subspace, capturing discriminative visual patterns that text semantics alone cannot represent. Experiments on CIFAR-100, ImageNet-100, ImageNet-R, and CUB-200 show state-ofthe-art performance, including gains of +2.7% on CIFAR-100 and +3.5% on ImageNet-100 over prior methods. The code will be included in the supplementary material.