Learning Consistent Temporal Grounding between Related Tasks in Sports Coaching

Video-LLMs often attend to irrelevant frames, which is espe-cially detrimental for sports coaching tasks requiring precise temporal lo-calization. Yet obtaining frame-level supervision is challenging: expensiveto collect from humans and unreliable from other models. We improvetemporal grounding without additional annotations during training byexploiting the observation that related tasks, such as generation andverification, must attend to the same frames. We enforce this via a self-consistency objective over select visual attention maps of tightly-relatedtasks. Using VidDiffBench, which provides ground-truth keyframe an-notations, we first validate that attention misallocation is a meaningfulbottleneck. We then show that training with our objective yields gains of+3.0%, +14.1% accuracy and +0.9 BERTScore over supervised finetun-ing across three sports coaching tasks: ExAct, FitnessQA, and ExpertAF.