Mitigate Modality-Asymmetric Forgetting via Stabilizing Visual Representations in CLIP-Based Class-Incremental Learning

Continual Learning (CL) enables models to learn sequentially while retaining prior knowledge. Recently, CL under the CLIP framework has gained increasing attention for its remarkable cross-task generalization capability. Despite recent progress, existing approaches overlook modality-asymmetric forgetting, where the visual modality undergoes severe degradation during CL. In this paper, we empirically identify this phenomenon and provide evidence that the lack of a stable structural anchor in the visual branch may drive high-level visual features to over-adapt to new tasks, thereby exacerbating catastrophic forgetting. To address this issue, we propose Visual Anchor-based Structural Transfer (VAST) to stabilize visual representations during continual learning. Specifically, we integrate a cross-layer representation enhancement module to construct the stable visual anchor, and further employ an information-theoretic structural distillation mechanism to transfer them across tasks, promoting stable adaptation and mitigating catastrophic forgetting. Extensive experiments across multiple benchmarks demonstrate that our method consistently surpasses SOTA approaches without relying on any replay data. Code is available at VAST.