Dual-Generalization-aware Minimization for Continual Fine-Tuning of Vision-Language Models

Continual fine-tuning of large vision–language models (VLMs)such as CLIP is crucial for adapting foundation models to evolving real-world distributions. However, such adaptation often disrupts pretrainedrepresentations and severely harms zero-shot generalization. Despite re-cent progress, existing methods overlook the underlying optimization ge-ometry, failing to anchor convergence within regions robust to distribu-tion shift. While recent studies suggest that flatter loss minima improvegeneralization, we empirically demonstrate that seeking local flatnesson a single task is insufficient to bridge the distributional gap betweenpre-trained and task-specific representations. Motivated by this obser-vation, we propose Dual-Generalization-aware Minimization (DGM), aplug-and-play optimization framework for continual adaptation of VLMs.DGM extends flatness-aware training by redefining the perturbation ob-jective to jointly promote pre-trained knowledge preservation and task-specific robustness. By incorporating these objectives into the inner max-imization step, DGM steers optimization toward solutions that remainstable under both zero-shot evaluation and task adaptation. Extensiveexperiments demonstrate that DGM effectively maintains zero-shot gen-eralization while achieving strong performance on sequential tasks.