Benchmarking MLLMs on Mistake Recognition and Explanation in Single-Step Components of Cooking

While mistake understanding in procedural video is advanc-ing toward increasingly complex multi-step tasks and diverse mistaketypes, it remains unclear whether current MLLMs can even recognize andexplain mistakes at individual step levels. This study proposes CookingMistake Recognition and Explanation (Cook-MRE), a dataset for assess-ing MLLMs in-depth performance in understanding cooking mistakeswithin each single-step. The evaluation uses an LLM-based approachalong two aspects: recognition (mistake presence and type classification)and explanation (error correction). By focusing on mistakes in the formof deviations from instructions, Cook-MRE reuses an existing cookingvideo dataset and synthesizes mistake samples through modification oftext instructions. It covers four basic operation elements and includes dif-ficulty labels based on visual distinguishability. Evaluation of 13 MLLMsshowed that our comprehensive metric, Full Pass Accuracy, was sub-stantially lower than mistake presence accuracy alone, indicating thedifficulty of explaining mistakes in detail even for single steps. Analysis byutilizing our type and difficulty labels showed notably low performanceon visually challenging problems, and scaling model size yields limitedimprovement on them. The dataset and code are publicly available athttps://github.com/Shun-Takashige/cook-mre-benchmark.