ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents

Tool-augmented vision–language models (VLMs) can solve multi-modal, multi-step tasks by calling external tools, yet they remain fragile in prac-tice. Existing works have two common gaps. Supervised fine-tuning (SFT) is builtmostly on successful trajectories and offers little signal for recovery after tool fail-ures, while sparse trajectory-level RL rewards provide limited guidance on whichstep failed and how to repair it. We introduce ReGRPO (Reflection-augmentedGroup Relative Policy Optimization), a framework that learns reflection-guidedcorrection in tool-using agents. ReGRPO starts with a structured reflective dataengine: we execute near-miss actions to collect grounded failure observations,then build Reflection-of-Thought triplets (ErrorType, Evidence, FixPlan) pairedwith corrected actions for warm-start SFT. We then optimize reflection tokens andcorrective actions jointly within local trajectories using group-relative advantages,and include a reflection-cost term to reduce unnecessary reflection. Experimentson GTA and GAIA show that, under the same backbone and tool suite, ReGRPOconsistently outperforms strong open-source baselines and achieves the best resultsamong the compared open-source controllers. Code and RoT data are available athttps://github.com/showlab/ReGRPO.