Preventing Expert Collapse in MoE-dVLMs via Modality-Wise Norm Alignment

Sparse Mixture-of-Experts (MoE) offers an attractive pathto scaling discrete diffusion vision–language models (dVLMs), but wefind that naïvely combining MoE with multimodal diffusion training canfail catastrophically. Specifically, MoE-dVLMs suffer from expert selec-tion collapse on visual tokens, where most image tokens are routed toonly a few experts and overall multimodal performance degrades sharply.We trace this failure to an upstream geometric mismatch at the modalityboundary rather than router optimization. Visual embeddings producedby deep Pre-Norm vision encoders have much larger norms than text em-beddings; when injected into a Pre-Norm language tower, the residualdynamics cause attention drowning and representational inertia, yield-ing highly concentrated, low-rank visual representations that are difficultfor the router to discriminate. To resolve this issue, we propose Tower-Align, a lightweight post-projector modality-wise norm alignment thatglobally rescales visual tokens to match text norm statistics while pre-serving their relative structure. TowerAlign stabilizes MoE routing andimproves multimodal accuracy with negligible overhead. On a practicalMoE-dVLM built from LLaDA-MoE-7B-A1B with a SigLIP2-SO400M-Patch14-384 vision encoder, TowerAlign delivers consistent gains acrossdiverse benchmarks and largely closes the gap to dense diffusion baselinesunder matched data budgets.