EgoEverything: A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment

Long-context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of its most important application domains. Nevertheless, the task remains highly challenging due to the need for reasoning over extended temporal contexts and diverse, unstructured activities. Although several benchmarks exist, most egocentric datasets rely on human-worn cameras and focus mainly on visual content, with limited consideration of underlying user behavior when forming video-related queries. EgoEverything is a benchmark that uses real gaze traces as a weak attention prior, rather than as a direct proxy for user intention, when generating questions. It comprises over 5,000 multiple-choice question-answer pairs, spanning more than 100 hours of video. By integrating measured gaze traces with a lightweight spatial sampling prior, it more faithfully captures natural AR-style querying behavior and offers a realistic evaluation setting for long-context egocentric video understanding in AR. We release our dataset at https://sai-lab-nyu.github.io/EgoEverything/.