CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

Human-human collaboration is a fundamental aspect of ev-eryday life, essential to success in a wide range of goal-directed activitiesfrom household tasks to professional teamwork. While much researchhas focused on modeling coordination and task execution, the cognitiveprocesses that support such collaboration, particularly Theory of Mind(the ability to infer others’ mental states), remain difficult to study innatural settings. To address this gap, we introduce a novel egocentric andexocentric video dataset capturing real-world collaboration in cookingscenarios. The dataset integrates multi-perspective video, high-qualityaudio, gaze tracking, and 3D scene and object scans, with annotations forshared gaze attention, social cues and interactions between agents, as wellas agent-object interactions. We establish benchmarks for Joint AttentionEstimation, Action Anticipation, and Collaborative Handover Prediction,enabling research on multimodal perception, proactive assistance, andcollaborative planning. By providing temporally aligned, richly annotatedmultimodal data, CoMind facilitates the development and evaluation ofAI systems capable of modeling complex social interactions and reasoningabout human behaviors in collaborative environments. Our dataset andbenchmarks are made available at https://comind.ethz.ch/.