Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

Existing jailbreak defense frameworks for Large Vision-LanguageModels (LVLMs) often suffer from a safety–utility tradeoff, where strength-ening safety inadvertently degrades performance on general visual-groundedreasoning tasks. In this work, we investigate whether safety and utilityare inherently antagonistic objectives. We focus on a modality-inducedbias direction consistently observed across datasets, which arises fromsuboptimal coupling between the LLM backbone and visual encoders.We further demonstrate that this direction undermines performance onboth tasks. Leveraging this insight, we propose TBOP (Two Birds, OneProjection), an efficient inference-time jailbreak defense that projectscross-modal features onto the null space of the identified bias direction toremove the corresponding components. Requiring only a single forwardpass, our method effectively breaks the conventional tradeoff, simultane-ously improving both safety and utility across diverse benchmarks.