Learning to Compose: Revisiting Proxy Task Design for Zero-Shot Composed Image Retrieval

Composed Image Retrieval (CIR) retrieves a target imagefrom a reference image and a textual modification. While supervisedCIR relies on costly triplets, Zero-Shot CIR (ZS-CIR) alleviates thisreliance through proxy tasks trained on image–text pairs. However, ex-isting proxy tasks primarily enhance visual and textual representationsto accommodate a predefined composition mechanism such as pseudo-word injection into a frozen text encoder or linear feature arithmetic.As a result, the composition function itself remains unlearned, limitingthe model’s ability to express diverse and fine-grained semantic mod-ifications. To address this, we propose FoCo, which models composi-tion as two coordinated stages: focusing on modification-relevant visualcontent, and then completing the target semantics. We realize thesethrough two proxy tasks: text-anchored visual aggregation to selectivelygather visual content guided by localized textual semantics, and context-conditioned semantic completion to transform these aggregated visualswith the remaining scene context into a coherent composed represen-tation. The tasks are trained jointly with a cross-instance contrastiveobjective, encouraging semantic diversity and discouraging shortcut com-position strategies. Extensive experiments on four ZS-CIR benchmarksshow FoCo’s state-of-the-art performance and improved generalization.