This Looks Distinctly Like That: Grounding Interpretable Recognition in Stiefel Geometry against Neural Collapse

Prototype networks provide an intrinsic case-based expla-nation mechanism, but their interpretability is often undermined byprototype collapse, where multiple prototypes degenerate to highly re-dundant evidence. We attribute this failure mode to the terminal dy-namics of Neural Collapse, where cross-entropy optimization suppressesintra-class variance and drives class-conditional features toward a low-dimensional limit. To mitigate this, we propose Adaptive Manifold Pro-totypes (AMP), a framework that leverages Riemannian optimizationon the Stiefel manifold to represent class prototypes as orthonormalbases and make rank-one prototype collapse infeasible by construction.AMP further learns class-specific effective rank via a proximal gradi-ent update on a non-negative capacity vector, and introduces spatialregularizers that reduce rotational ambiguity and encourage localized,non-overlapping part evidence. Extensive experiments on fine-grainedbenchmarks demonstrate that AMP achieves state-of-the-art classifica-tion accuracy while significantly improving causal faithfulness over priorinterpretable models.