Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning

Multimodal large language models (MLLMs) can inadver-tently memorize privacy-sensitive information during training. Whileexisting unlearning methods can remove such content, they often severelydegrade the model’s foundational capabilities, such as general imageunderstanding. This critical shortfall motivates our investigation into be-nign memory forgetting, the precise removal of targeted, privacy-sensitiveknowledge while rigorously preserving unrelated capabilities. To pioneerand evaluate progress toward this objective, we introduce S-MLLMUnBench, the first benchmark designed to jointly and quantitatively assessan unlearning method’s efficacy in knowledge erasure and the preservationof image understanding. Furthermore, we propose the Sculpted MemoryForgetting Adapter (SMFA), a new framework that enables benign mem-ory forgetting. SMFA confines forgetting to designated memory regions,maintaining overall model performance. By initially fine-tuning the modelto replace sensitive outputs with refusals, SMFA generates a memoryforgetting adapter, followed by a retaining anchor-guided masking mech-anism that safeguards unrelated knowledge. Extensive experiments onS-MLLMUn Bench demonstrate that existing methods fail to achievebenign forgetting, whereas our proposed SMFA serves as an effectivebaseline, successfully achieving targeted knowledge erasure without com-promising the model’s foundational visual capabilities. Code and dataare available at https://github.com/zeng-zhen/S-MLLMUn.