Learn to Rank: Visual Attribution by Learning Importance Ranking

Interpreting the decisions of complex computer vision modelsis crucial to establish trust and accountability, especially in safety-criticaldomains. An established approach to interpretability is generating visualattribution maps that highlight regions of the input most relevant to themodel’s prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biasedand architecture-specific. Meanwhile, perturbation-based methods arecausally grounded, yet they are expensive and for vision transformersoften yield coarse, patch-level explanations. Learning-based explainersare fast but usually optimize surrogate objectives or distill from heuris-tic teachers. We propose a learning scheme that instead optimizes dele-tion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learn-ing and replace the hard sorting with a differentiable relaxation usingGumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our methodproduces dense, pixel-level attributions in a single forward pass with op-tional, few-step gradient refinement. Our experiments demonstrate con-sistent quantitative improvements and sharper, boundary-aligned expla-nations, particularly for transformer-based vision models. Code and pre-trained models are available at https://github.com/dschinagl/AHA.