Curvature-Guided Mixing for MLLM Adaptation

Fine-tuning Multimodal Large Language Models (MLLMs)on specialized tasks often leads to catastrophic forgetting of their gen-eral capabilities. Existing model merging methods to combat this areoften heuristic or use sub-optimal objectives. We propose Curvature-Guided Mixing (CGM), a theoretically grounded framework that mergespre-trained and fine-tuned models. CGM formulates a joint optimiza-tion objective and uses a second-order (Hessian) approximation of theloss landscapes to analytically derive an optimal, closed-form “soft mix-ing” ratio. This ratio intelligently blends parameters based on their rel-ative task-specific curvatures. We also introduce CGM† , a robust “hardmixing” variant that performs sparse parameter selection guided by anovel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen-2.5VL across multiple downstream tasks show that CGM and CGM†consistently improve the trade-off between task specialization and gen-eral knowledge retention over existing methods. Code is available atgithub.com/zzsyjl/CGM-ECCV-2026.