Gradient sparsity regularization for training unlearning-compatible models

In this paper, we approach the machine unlearning (MU)problem from a novel perspective: Can we design a learning strategythat simplifies the later MU problem, i.e., leads to unlearning-compatiblemodels? Motivated by recent findings that the cost to reverse SGD (un-learning error) is proportional to the accumulated loss curvature and thatthe generalization error (a measure of acquired subject-specific knowl-edge) is bounded by loss sharpness, we focus on the loss sharpness as thekey factor to train unlearning-compatible models. By studying the theo-retical implications of defining the sharpness over different perturbationnorms, we propose a variation of the sharpness, quantified by the l1 normof the gradient, which minimizing it leads to a flatter learning trajectoryand, as a result, smaller unlearning and generalization errors. Consider-ing these results, to have unlearning-compatible models, we propose toregularize the training loss with l1 norm of the gradient, which encour-ages gradient sparsity during training. Our experiments show that onlyby using our regularizer in the training stage, the MU performance ofvarious baselines is considerably improved, suggesting the effectivenessof our approach in simplifying the MU problem by training unlearning-compatible models. Moreover, we demonstrate that the benefits of ourapproach become even more significant when dealing with poisoned datain backdoor attack scenarios. Our investigations show that in such ascenario, gradient sparsity not only simplifies model cleaning but alsodamps the gradient spikes, which are typically caused by poisonous dataand encourage the model to automatically learn less from such poisonousdata points.