REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency

Knowledge Distillation (KD) transfers knowledge from a largeteacher model to a smaller student by aligning their predictive distri-butions. However, conventional KD formulations - typically based onKullback–Leibler divergence - assume that the teacher provides reliabletargets. In practice, teacher predictions are often noisy or overconfident,and existing correction-based approaches rely on ad-hoc heuristics andextensive hyper-parameter tuning, which hinders generalization. We in-troduce REDistill (Robust Estimator Distillation), a simple yet prin-cipled framework grounded in robust statistics. REDistill replaces thestandard KD objective with a power divergence loss, a generalizationof KL divergence that adaptively downweights unreliable teacher out-put. This formulation provides a unified and interpretable treatmentof teacher noise, requires only logits, integrates seamlessly into existingKD pipelines, and incurs negligible computational overhead. Extensiveexperiments on CIFAR-100 and ImageNet-1k demonstrate that REDis-till consistently improves student accuracy in diverse teacher-student ar-chitectures. Remarkably, it achieves these gains without model-specifichyper-parameter tuning, underscoring its robustness and strong general-ization to unseen teacher–student pairs.