Open Your Eyes: Benchmarking the Detection of Fabricated Realities and Weaponized Ethics in VLMs

As Vision–Language Models (VLMs) increasingly power au-tonomous agents—from coding assistants that interpret terminal outputand execute shell commands, to Computer-Use Agents with full desktopcontrol, to HR screening and loan evaluation bots—these agents act di-rectly over visual artifacts from untrusted sources. While prior work hasstudied direct multimodal jailbreaks, text-based indirect prompt injec-tion, and HTML-embedded attacks, the security implications of Multi-modal Indirect Prompt Injection (M-IPI) in realistic agentic workflowsremain insufficiently understood. We ask a central question: does pro-cessing an artifact visually, rather than as text, degrade an agent’s abil-ity to resist embedded attacks? To study this, we introduce the M-IPIBenchmark—a suite of 2,600 high-fidelity visual artifacts encompass-ing two attack families: (1) Technically-Framed Attacks, where maliciouscommands are interwoven with genuine debugging workflows in terminalscreenshots, and (2) Ethics-Framed Attacks, a novel vector where adver-saries exploit alignment priors such as fairness mandates to override eval-uation policies—and use it to systematically analyze modality-dependentfailures. We evaluate five model families in paired vision and text-onlyconfigurations that receive identical content, isolating the modality effect.Our results reveal that VLMs consistently underperform text-only coun-terparts at detecting embedded attacks—a “visual authority” effect whererendered presentation suppresses critical evaluation. For end-to-end at-tack success, both modalities show comparable susceptibility (∼50% fortechnical, >80% for ethics-framed), with qualitatively distinct failureprofiles: VL models disproportionately trust visually-rendered commandswhile text-only models are more vulnerable to natural-language direc-tives. These findings indicate that the vulnerabilities are rooted in theshared language backbone and that alignment priors represent a dis-tinct, exploitable attack surface. Our benchmark artifacts are releasedunder gated, research-only access at https://github.com/adityakm24/OpenYourEyes to support future defensive research.