SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos

To enable personalized, real-time coaching using AugmentedReality glasses or fixed camera setups in domains such as sports, cooking,or music, a system must understand not just what a person does, buthow well they execute an activity. In an ego-exo video setting, this re-quires simultaneously detecting individual skilled actions and classifyingeach as correct or needing improvement, which Ego-Exo4D’s proficiencydemonstration benchmark formalized. We first adapt seven state-of-the-art temporal action detection architectures to this task, extend the eval-uation protocol to disentangle detection from grading, and show that ex-isting methods grade near-randomly. We then introduce SkillSpotter , apose-aware multi-view architecture that jointly detects and grades skilledactions through three task-specific modules: (1) adaptive temporal sup-pression to handle the varying density of skilled actions across diverseactivities, (2) gated 3D body pose fusion to leverage body kinematics as acomplementary signal to visual features, and (3) bidirectional cross-viewattention to combine ego and exo views e!ectively. SkillSpotter improvesclass-specific mAP from 12.40 to 21.82 (+76%) and balanced accuracyfrom 55.99% to 60.40% over the best baseline. SkillSpotter ’s modulestransfer to other temporal action detection models with consistent gainsand our method generalizes beyond Ego-Exo4D to HoloAssist.Code: https://github.com/eth-siplab/SkillSpotter