Adapting MLLMs for Nuanced Video Retrieval

Our objective is to build an embedding model that capturesthe nuanced relationship between a search query and candidate videos.We cover three aspects of nuanced retrieval: (i) temporal, (ii) negation,and (iii) multimodal. For temporal nuance, we consider chiral actionsthat need distinguishing between temporally opposite actions like “open-ing a door” vs. “closing a door”. For negation, we consider queries withnegators such as “not”, “none” that allow a user to specify what theydo not want. For multimodal nuance, we consider the task of composedretrieval where the query comprises a video along with a text edit in-struction. The goal is to develop a unified embedding model that handlessuch nuances effectively. To that end, we repurpose a Multimodal LargeLanguage Model (MLLM) trained to generate text into an embeddingmodel. We fine-tune it with a contrastive loss on text alone with carefullysampled hard negatives that instill the desired nuances in the learned em-bedding space. Despite the text-only training, our method achieves stateof the art performance on all benchmarks for nuanced video retrieval. Wealso analyze how this improvement is achieved, and show that text-onlytraining reduces the modality gap between text and video embeddingsleading to better organization of the embedding space.