Prefill-Time Interventions against Adversarial Attacks on Large Vision-Language Models

Recent studies have revealed critical vulnerabilities in Large Vision-Language Models (LVLMs) to sophisticated adversarial attacks that induce the model into generating adversary-specific content. However, existing LVLM-oriented detectors largely rely on external judging pipelines that incur substantial computational overhead, precluding their use in real-time scenarios. Towards this end, we propose PTI (PrefillTime Intervention), a method that probes an LVLM’s internal states to detect adversarial images before any response token is generated. PTI combines two complementary modules trained only on benign data: (i) a masked reconstruction module in the vision tower that identifies visual anomalies through token-wise reconstruction errors, and (ii) a dynamic modeling module in the language model backbone that tracks layer-wise trajectories of salient tokens to detect semantic inconsistencies. We establish a comprehensive evaluation framework covering both black-box and white-box attacks, including L -bounded, unrestricted, p and patch-based perturbations. Extensive experiments demonstrate that PTI achieves state-of-the-art detection performance and robust generalization across diverse threat models. Crucially, its minimal inference overhead paves the way for secure, real-time LVLM applications.