LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

Visual document retrieval requires rapidly locating relevantpages from large multi-modal corpora in response to user queries. Whilerecent methods powered by Multi-modal Large Language Models (MLLMs)show competitive accuracy, they suffer from prohibitive computationalcosts by applying intensive MLLM encoding to every single page. Mean-while, we observe that user queries are typically keyword-anchored, con-taining semantically rich words that are expected to appear directly inthe visible text of relevant pages, offering an efficient cue for quickly nar-rowing down candidate pages. Building on this insight, we propose Light-STAR, an efficient framework that decomposes visual document retrievalinto: 1) LLM-free Visual Selection, which utilizes content-grounded queryencoding to focus on informative words and employs LLM-free visual em-beddings to produce a high-recall candidate set; and 2) Vision-adaptiveSemantic Refinement, which further performs fine-grained semantic match-ing exclusively on these top candidates via adaptive region-wise featurefusion to effectively combine textual and layout cues, optimized through ahardness-aware contrastive objective. Experimental results demonstratethat LightSTAR achieves state-of-the-art retrieval accuracy while reduc-ing end-to-end latency by several-fold, offering a highly practical solutionto the accuracy-efficiency trade-off in visual document retrieval. Code isavailable at https://github.com/bokufa/LightSTAR.