Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval

Zero-shot video moment retrieval (ZMR) aims to overcomethe limitations of traditional approaches that require large-scale datasetsannotated with text and its relevant temporal spans. Despite advances inpre-trained vision–language models (VLMs) and multimodal large lan-guage models (MLLMs), existing ZMR methods still heavily depend onquery-to-video content similarity, making them vulnerable to modal-ity and language-style gaps. These gaps lead to unreliable span pro-posals and unstable moment retrieval results. To address this issue, wepropose Self-Similarity-based Moment proposal and Scoring (Self-SiMS)that instead exploits intrinsic relationships within videos, enabling ro-bust span generation and scoring. By deriving self-similarity only fromthe video content, we circumvent the noisy and mismatched patternsof query–frame or query–caption similarities, thereby mitigating bothmodality and language-style gaps. Furthermore, we introduce a query-aware MLLM-based reasoning stage to further sharpen alignment be-tween text and video. Extensive experiments demonstrate that Self-SiMSachieves the state-of-the-art performance across ZMR benchmarks.