Continuous Heart Rate Variability Estimation from Egocentric Systems for Skill Assessment

Egocentric vision systems capture human behavior from visi-ble cues, but overlook physiological indicators of autonomic states suchas stress, engagement, and attention. Heart rate variability (HRV) is awidely used noninvasive marker of autonomic regulation under stress.HRV reflects small timing differences between successive heartbeats andhas so far been out of reach for egocentric platforms, where motion andnoise in gaze video mask exactly this fine-grained timing. We proposeEgoHRV, a method that estimates HRV as well as heart rate (HR) fromthe gaze cameras that are already integrated into egocentric headsets. Ourpipeline combines a 3D backbone with a novel low–high decompositionmodule that extracts the blood volume pulse (BVP) signal from gazevideo. Our cross-domain pretraining aligns the frequency-domain repre-sentations of contact-based and camera-derived signals. This alignmentgives EgoHRV the temporal precision to recover HRV from the subtlefluctuations in gaze video. EgoHRV achieves state-of-the-art accuracy forHR and HRV estimation from egocentric video, and its uncertainty-awaredesign improves downstream behavioral modeling. Integrating our HRVestimates and confidence measures into EgoExo4D’s proficiency estimatorraises accuracy by 17.8%. Beyond skill, continuous HRV estimation alsoopens egocentric systems to stress- and arousal-aware estimation tasks.Code: https://github.com/eth-siplab/EgoHRV