GCMRD: Global Consistency Multi-teacher Robustness Distillation

Adversarial Robustness Distillation (ARD) aims to trans-fer both natural and robust knowledge from a strong teacher modelto a lightweight student model, thereby achieving robust performancein resource-constrained settings. The dual-teacher robustness distilla-tion paradigm further provides an inspiring solution for optimizing the‘accuracy-robustness’ trade-off. However, existing dual-teacher distilla-tion frameworks fail to fully align the supervision provided by teachergroup, resulting in suboptimal performance. To fully utilize the poten-tial of dual-teacher robustness distillation, we propose a novel multi-teacher framework called global consistency multi-teacher robustness dis-tillation (GCMRD), which improves the existing dual-teacher paradigmfrom three perspectives. First, for internal maximization, we introducean integrated soft-label attack objective to replace hard-label or single-teacher-guided objective, leading to consistent attack preferences acrossboth natural and robust teachers. Second, for external minimization, wedesign a fully-repulsion regularization mechanism based on the erroneousadversarial outputs of the natural teacher, which enables more compre-hensive utilization of teacher supervision. Finally, we employ the outputsof the natural teacher to guide the fine-tuning of the robust teacher, in-directly balancing the student model’s accuracy-robustness trade-off byenforcing consistency with the teacher ensemble. Experimental resultsdemonstrate that our GCMRD achieves state-of-the-art performance inboth natural accuracy and adversarial robustness.