Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models

Sparse autoencoders (SAEs) have recently been proposed asinterpretable tools for concept-level manipulation, under the assumptionthat isolated features can serve as controllable intervention points. In thiswork, we systematically evaluate this assumption in the context of ob-ject erasure and steering in diffusion models. We show that while SAEsreliably detect and localize semantic concepts within diffusion modelactivations, direct intervention in their latent space frequently inducesout-of-distribution activations, resulting in severe visual artifacts. To dis-entangle detection from intervention, we use SAE activations purely assemantic detectors to identify image regions containing the target object,and replace those patch embeddings with the ones that do not containit. This detection-based replacement preserves the diffusion model’s ac-tivation statistics and produces significantly cleaner erasure results thanlatent steering. Our findings reveal a fundamental gap between conceptdetection and concept intervention in diffusion models: monosemanticor sparse features are not inherently suitable as control knobs for steer-ing. These results position SAEs as powerful interpretability tools foranalyzing generative models, but highlight important limitations whenused for direct manipulation, such as unlearning. We release the code athttps://eidoslab.github.io/PER/.