The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models

Safety alignment of text-to-image (T2I) diffusion models aimsto suppress harmful generations while preserving utility on benign prompts.Recent methods often appear to deliver high safety with high utility, butthis conclusion rests largely on coarse global utility metrics (e.g., FID,CLIPScore) that are insensitive to fine-grained semantic correctness, cre-ating an illusion of high utility. We show that when utility is measuredwith structured evaluation, this illusion breaks: on TIFA (Text-to-ImageFaithfulness evaluation with Question Answering), safety-aligned modelssuffer substantial drops in semantic fidelity, including failures in objectcounts, attributes, and relationships. To diagnose the source of this gap,we analyze the text-encoder prompt embedding space and uncover se-mantic collapse, a contraction of embedding spread coupled with distor-tion of inter-prompt similarity structure, which strongly correlates withstructured utility loss. Guided by this insight, we propose Structure-Aware Geometric Regularization (SAGE)1 , a safety alignment objec-tive that explicitly preserves embedding spread and inter-prompt rela-tional structure during adaptation. Our method restores structured util-ity (TIFA +5.0% over prior state-of-the-art) while maintaining strongsafety performance and competitive coarse-grained utility scores.