AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs

Inspired by text prompts in large language models, visualprompts have been explored to enhance the perceptual capabilities oflarge vision-language models (LVLMs). However, performance tends tosaturate under single visual prompt designs, making further prompt engi-neering increasingly ineffective. To address this limitation, we shift fromprompt engineering to prompt retrieval and propose AutoV, a lightweightframework for instance-adaptive visual prompt identification. Given aninput image and a textual query, AutoV automatically locates the mostsuitable visual prompt from a diverse candidate pool. Training such a re-trieval framework requires prompt-level supervision, yet prompt qualityis inherently ambiguous and difficult to assess reliably, even for humans.To enable automatic supervision, we evaluate visual prompts using a pre-trained LVLM and label them according to their prediction losses. Usingthe loss-oriented ranking as a robust training signal, AutoV learns to re-trieve the query-aware optimal prompt for each instance without manualannotation. Experiments indicate that AutoV enhances the performanceof various LVLMs on image understanding, captioning, grounding, andclassification tasks. For example, AutoV improves LLaVA-OV by 10.2%on VizWiz and boosts Qwen2.5-VL by 3.8% on MMMU, respectively.