WildWorld: A Large-Scale Dataset for Action-Conditioned World Modeling with Explicit State Annotations

Dynamical systems theory and reinforcement learning viewworld evolution as the dynamics of latent states driven by actions, withvisual observations providing partial information about those states. Re-cent video world models attempt to learn these action-conditioned dy-namics from data. However, existing datasets rarely meet these require-ments: they typically lack diverse and semantically meaningful actionspaces, and actions are directly tied to visual observations rather thanmediated by underlying states. As a result, actions are often entangledwith pixel-level changes, making it difficult for models to learn structuredworld dynamics and maintain consistent evolution over long horizons. Inthis paper, we propose WildWorld, a large-scale action-conditioned worldmodeling dataset with explicit state annotations, automatically collectedfrom a photorealistic AAA action role-playing game (Monster Hunter:Wilds). WildWorld contains over 108 million frames and features morethan 450 actions, including movement, attacks, and skill casting, togetherwith synchronized per-frame annotations of character skeletons, worldstates, camera poses, and depth maps. We further derive WildBench toevaluate models through Action Following and State Alignment. Exten-sive experiments reveal persistent challenges in modeling semanticallyrich actions and maintaining long-horizon state consistency, highlightingthe need for state-aware video generation.