ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling

Multi-objective masked image modeling (MIM) combines com-plementary learning signals (token distillation, CLS alignment, and pixelreconstruction) but existing methods weight these objectives with globalscalars, ignoring spatial heterogeneity across patches. We present Ex-PLoRe (Expert Patch-Level Loss Routing), which repurposes Soft Mix-ture of Experts (MoE) dispatch weights as learned, per-patch loss coef-ficients. The key mechanism is loss-coupling: allowing loss gradients toflow through dispatch weights to the router enables content-dependentspecialization, where different patches receive different emphases acrossobjectives. A detach ablation confirms loss-coupling as the core mecha-nism, degrading performance by 1.6% when gradients are blocked. OnImageNet-1K with ViT-Base, ExPLoRe improves over non-MoE base-lines on two objective combinations (Token+CLS: +0.5% k-NN, +4.4%linear probe; Token+Pixel: +2.2% k-NN), achieving 80.6% linear probeand 85.3% finetuning accuracy, competitive with published methods. Fordownstream transfer, we develop adaptation recipes (Freeze Routing, Ex-pert Dropout, and Freeze Attention) that improve MoE finetuning by+1.5% over the vanilla MoE, and close a 2.5–2.9 mIoU segmentation gapso that MoE models match or exceed non-MoE baselines on ADE20K.