SFKD: Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction

Most existing knowledge distillation methods focus on ho-mogeneous models (e.g., CNN→CNN), thereby overlooking the flexi-bility and potential of knowledge transfer across heterogeneous mod-els. Due to intrinsic inductive bias discrepancies between heterogeneousmodels that cause spatial distribution inconsistencies, prior heteroge-neous distillation methods often weaken or discard spatial informationin heterogeneous representations. However, the spatial information inrepresentations often encodes transferable global structural semantics aswell as architecture-specific local details, and therefore should not bedirectly ignored. To better leverage the spatial information encoded inheterogeneous representations, we propose a Spatial–Frequency Joint-Aware Heterogeneous Knowledge Distillation framework (SFKD). Byleveraging the complementary properties of wavelet transform spatiallocality and Fourier representations in characterizing global energy dis-tributions, we first apply multi-level discrete wavelet transform to ex-plicitly decouple spatial information. The resulting wavelet sub-bandsare further refined by a dual-stream dual-stage refinement module, andfinally combined with a Gaussian-filtered frequency loss to selectivelycapture informative global information. Extensive experiments on mul-tiple benchmark datasets under both homogeneous and heterogeneousmodels demonstrate the superiority of our method. Code is available athttps://github.com/cpcpWang/SFKD.