Learning Sample-wise Rank-Aware Interpolation Weights for Composed Visual Data Retrieval

At the heart of composed visual data retrieval is the fusionof a reference visual input and a textual modification into a single query.While current state-of-the-art methods utilize multimodal large languagemodels for this fusion, their complexity introduces prohibitive query-time latency, limiting their scalability. We instead revisit the efficacy ofsimple linear interpolation within an embedding space, and introduceSRAIN, the first framework that dynamically predicts query-specific in-terpolation weights. The key challenge lies in the fact that the qualityof an interpolation weight should be measured by the interpolated em-bedding’s discriminability from negatives as well as its proximity to truetargets; this makes collecting and predicting optimal weights intractable.We overcome this bottleneck through two key innovations: batch-wiserank-aware weight estimation during training, and a compact memorybank that synthesizes hard negatives during inference. SRAIN achievesthe best in composed video retrieval and matches the current state of theart in composed image retrieval, all while substantially reducing query-time latency compared to MLLM-based alternatives.