WALL-EVE: World Alignment with Rule Learning in Visual Environments

Constructing world models for AI agents in visual environ-ments has long been a significant challenge, chiefly due to the di!cultyof achieving precise alignment of the dynamics between the world modeland the environment, given noisy pixel-level observations. Existing worldmodels solely rely on video or image generation, which are exceedinglycomplex and expensive to train, yet still su"er from hallucinations andviolations of basic rules due to misalignment. In this paper, we show thata few rules su!ce to convert a pretrained large language model (LLM)to an accurate world model for a specific visual environment. We pro-pose a training-free approach to e!ciently learn these complementaryrules from multimodal agent trajectories collected in the visual environ-ment. In particular, we convert the multimodal input into structuredtext and extract symbolic rules using an LLM by comparing agents’ truetrajectories with world model predictions if they have conflicts. We usethis neurosymbolic approach to induce, update, and prune rules dur-ing exploration, resulting in a precise world model. It allows us to builda novel model-predictive control (MPC) agent, “WALL-EVE”, with-out training any policy. Before taking each action, WALL-EVE utilizesan LLM as a look-ahead optimizer to determine the next k-step actionsthrough interactions with our world model. On challenging tasks in ALF-World and Minecraft, WALL-EVE achieves higher success rates thanexisting methods, while reducing inference times and the number of to-kens required for reasoning. In ALFWorld, WALL-EVE surpasses thestate-of-the-art method RAFA with only 17% of RAFA’s token usage.In Minecraft, WALL-EVE outperforms baselines by 8-30% in successrate but costs 8–20 fewer replanning rounds and only 60–85% tokens.Code is available here.