StructPolicy: Structure-Guided Imitation Learning Robust to Visual Domain Shifts

Imitation Learning (IL) offers an effective approach for robotmanipulation by learning a mapping from visual inputs to actions. How-ever, this paradigm suffers from poor robustness under visual domainshifts (e.g., lighting conditions, camera viewpoints, etc.), often failingto perform manipulation. Our key insight in tackling this problem is toconstruct a Structure Map encoding the fine-grained object structuresthat remain invariant across visual domains and vital for manipulation.Based on this insight, we propose StructPolicy, a method elaboratelydesigned to incorporate the domain-invariant Structure Map into the ILpolicy through two modules: StructCon and StructEncoder. StructCon isan automated Structure Map construction module that leverages struc-ture primitives to enable flexible transformation and composition to forma wide range of objects. StructEncoder is a hierarchical network thatefficiently captures structural relationships and affordance from Struc-ture Map. This provides the IL policy with both domain-invariant andstructurally-aware features, guiding it toward robust manipulation un-der visual domain shifts. We extensively evaluate StructPolicy across 49manipulation tasks in multiple benchmarks and diverse real-world tasksunder various visual changes. The results demonstrate consistent andsignificant performance improvements across all tasks, validating thatStructPolicy enhances the effectiveness and robustness against visual do-main shifts of the IL policy, improving manipulation accuracy.