Twin-DAgger: Synergizing Digital Twins and Human Corrections for Efficient Robot Manipulation

Imitation learning enables robots to acquire complex manipulation skills from expert demonstrations, but offline-trained policies struggle to generalize beyond their training distribution, where small errors compound into severe distribution shift. Dataset Aggregation (DAgger) mitigates this by collecting human corrections during deployment, yet conventional DAgger operates entirely in the real world with a oneto-one ratio between expert effort and training trajectories makes comprehensive out-of-distribution (OOD) coverage impractical. Digital Twin (DT) techniques can scale data generation, but when applied to DAgger data without modification, they ignore the spatial and temporal failure patterns that DAgger rollouts reveal, leaving policies vulnerable precisely where and when they fail most. We propose Twin-DAgger, which treats each real-world intervention as simultaneously a spatial probe identifying failure-prone workspace regions and a temporal prior marking critical deviation moments. Twin-DAgger exploits these signals through two targeted augmentation mechanisms: difficulty-based stratified sampling concentrates synthetic data generation on high-failure-rate regions, and intervention-focused trajectory warping perturbs trajectories at each intervention onset to simulate a distribution of near-failure states while preserving the expert’s corrective recovery. On real-world manipulation tasks, Twin-DAgger matches conventional DAgger performance using only 15% human interventions and outperforms failure-agnostic DT baselines by 20% success rate under equal data budgets, demonstrating that generating data where and when the policy fails, rather than generating more data uniformly, is the critical driver of sample-efficient robustness.