Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection

Existing red-teaming studies on GUI agents face two funda-mental limitations: adversarial perturbations require white-box accessunavailable in commercial deployments, while prompt injection is increas-ingly neutralized by stronger safety alignment. To study robustness undera more practical threat model, we propose Semantic-level UI ElementInjection, a black-box red-teaming paradigm that overlays safety-alignedand harmless UI elements onto screenshots to misdirect the agent’s visualgrounding. Our method couples a modular Editor–Overlapper–Victimpipeline with iterative search that samples multiple candidate edits, keepsthe best cumulative overlay, and adapts future prompt strategies basedon previous failures. Experiments across 19 victim models spanning 8model families show that strategic optimization substantially outperformsrandom injection (3.5–6.9× on the most robust victims) and transfersnear-perfectly across architectures, confirming model-agnostic visual-semantic vulnerabilities. After the first successful attack, the victim stillclicks the attacker-controlled icon in over 15% of subsequent independenttrials versus below 1% for random injection, establishing that strategicallyplaced icons act as persistent attractors that causally redirect groundingrather than introducing incidental clutter. Public code is available athttps://github.com/HashTAG00002/UI-Injection.