AffoGato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale

Affordance grounding aims to localize where to interact withan object, a fundamental capability for embodied agents. Yet progressis bottlenecked by data: manual annotation is prohibitively expensiveand confines existing datasets to a narrow set of predefined objectand affordance categories. We introduce Affogato, a framework foropen-vocabulary affordance grounding centered on Affogato-750K, alarge-scale dataset of 750K 3D affordance heatmaps paired with naturallanguage queries. We build it with a fully automated pipeline that or-chestrates foundation models to generate them at scale without humanlabeling. It covers significantly more diverse categories than any existingdataset. For reliable evaluation, we further provide 5K human-verified testpairs. We also present Espresso-3D and Espresso-2D, simple yet effec-tive models with a unified architecture across both modalities. Pretrainingon Affogato-750K improves both Espresso and prior methods and yieldsthe largest gains on unseen object and affordance categories, showingthat it provides broadly transferable supervision across architectures.