Residual-Guided Expert Specialization for Incomplete Multimodal Learning

As real-world prediction systems often face missing modali-ties at inference, incomplete multimodal learning (IML) remains a prac-tical challenge. While prior methods aim to learn representations robustto missing inputs, representations from incomplete modalities inevitablydeviate from their full-modality counterparts due to missing evidence. Toexplicitly leverage these deviations, we propose MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts frameworkthat guides expert specialization based on how representations are re-shaped by missingness. By contrasting task representations derived fromincomplete inputs with their complete counterparts during training, wederive a privileged residual signal that captures this representational gap.The residual signal guides a residual router to assign samples to the ex-perts specialized for the corresponding deviation patterns. In parallel, afeature router learns to imitate this routing behavior using only incom-plete inputs, enabling deployment without access to full modalities. Tomitigate this train–test router gap, we develop a discrepancy-aware noiseregularization that adaptively perturbs the residual router’s decisionswhen the feature router deviates, enhancing the expert robustness underimperfect imitation. Experiments on multimodal classification (CASIA-SURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) un-der missing scenarios show that MARS consistently surpasses baselines,while remaining efficient and extensible to diverse backbones and tasks.