RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models

Region-level retrieval aims to align user-specified image regions with relevantregions or textual descriptions, playing a crucial role in realworld applications such as e-commerce product search and RAG. Although recent Large Multimodal Models (LMMs)have made significant strides in multimodal retrieval, they primarily focus on global-level tasksand struggle to capture effective region-level representations. To bridge this gap, we presentRegRet, an LMM-based Region-level Retrieval framework that enhances the regional rep-resentations without compromising overall global retrieval performance. At its core, RegRetintegrates a Region-Aware Encoder to capture detailed regional features while balancing themwith the global background context. To further enhance the fine-grained understanding anddiscriminability of representations, we design a multi-stage training pipeline that includesdetailed localized captioning and regional contrastive learning tasks. In addition, consideringthe absence of region-level contrastive training data and the limited diversity of evaluationtasks in current benchmarks, we introduce the REGMB benchmark. It comprises 225k con-trastive pairs, covering four multimodal retrieval tasks. Extensive experiments validate theeffectiveness of our approach. RegRet outperforms strong baselines in the zero-shot setting.Further training with contrastive learning leads to an average improvement of more than 20%on both REGMB and public benchmarks, while achieving comparable or better results onglobal-level retrieval tasks. The code and data will be released for future research.