Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models

Large vision-language models (LVLMs) are vulnerable to ty-pographic attacks, where misleading text inserted into an image can over-ride visual understanding. However, existing evaluation protocols anddefenses are largely focused on object recognition and do not considertext-reading capability. This is a critical oversight: real-world scenariosoften require both recognizing objects and reading scene text (e.g., rec-ognizing pedestrians while reading tra!c signs), where simply ignoringall text for robustness is unacceptable in practice. To address this gap,we introduce a novel task, Read-or-Ignore VQA (RIO-VQA), whichjointly evaluates both requirements: models must decide, from context,when to read scene-text and when to ignore inserted distractor text. Toevaluate this capability, we present RIO-Bench, a same-scene counter-factual benchmark that holds the scene fixed while varying only questionintent (object vs. text) and text condition (clean vs. attack), enablingdirect comparisons of model behaviors with reduced confounding fac-tors. Using RIO-Bench, we highlight a trade-o": representative defensesdeveloped in object-centric settings can achieve robustness by suppress-ing text sensitivity, at the cost of text-reading performance (i.e., “ignor-ing” text). Motivated by this trade-o", we provide a data-driven defensebaseline that improves both requirements on RIO-Bench, complementingprior text-ignoring baselines. Overall, this work highlights a fundamen-tal misalignment between the current object-centric robustness scope andreal-world multimodal requirements, providing a principled path towardreliable LVLMs.