Rethinking Robust Adversarial Concept Erasure in Diffusion Models

Concept erasure methods aim to remove specific unsafe tar-get concepts in diffusion models while preserving image generation utility.To address the vulnerability that erased concepts can be easily recoveredunder adversarial attacks, adversarial concept erasure methods integrateadversarial optimization into the concept erasure process. However, exist-ing adversarial concept erasure methods face a trade-off between robust-ness and computational cost. We attribute this to adversarial optimiza-tion techniques that use random samples to approximate the adversarialobjective function. Adversarial optimization that uses a small numberof samples fails to produce adversarial embeddings that accurately cap-ture the target concept space. To mitigate this limitation, we proposeSemantic-Guided Adversarial Optimization, which uses a single sampleto produce adversarial embeddings that better capture the target con-cept space. We also propose Semantic-Guided Concept Erasure, whichautomatically maps the target concept to a semantically similar surro-gate. Extensive experiments on not-safe-for-work content, artistic styles,and object-related concepts demonstrate that our method, S-GRACE(Semantic-Guided Robust Adversarial Concept Erasure) achieves state-of-the-art erasure robustness and superior image generation utility, withsignificantly lower computational cost than existing methods. Our codeis available at https://github.com/Qhong-522/S-GRACE.