Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision

Traditional Visual Grounding (VG) predominantly relies ontextual descriptions to localize objects, a paradigm that inherently strug-gles with linguistic ambiguity and often ignores non-verbal deictic cuesprevalent in real-world interactions. In natural egocentric engagements,hand-pointing combined with speech forms the most intuitive referringmechanism. To bridge this gap, we introduce EgoPoint-Ground, thefirst large-scale multimodal dataset dedicated to egocentric deictic visualgrounding. Comprising over 15k interactive samples in complex scenes,the dataset provides rich, multi-grained annotations including hand-targetbounding box pairs and dense semantic captions. We establish a com-prehensive benchmark for hand-pointing referring expression resolution,evaluating a wide spectrum of mainstream Multimodal Large LanguageModels (MLLMs) and state-of-the-art VG architectures. Furthermore, wepropose SV-CoT, a novel baseline framework that reformulates groundingas a structured inference process, synergizing gestural and linguistic cuesthrough a Visual Chain-of-Thought paradigm. Extensive experimentsdemonstrate that SV-CoT achieves an 11.7% absolute improvement overexisting methods, effectively mitigating semantic ambiguity and advanc-ing the capability of agents to comprehend multimodal physical intents.Code: https://github.com/lingli1724/Egopoint