DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models

As Vision-Language Models (VLMs) become increasingly so-phisticated and widely used, it becomes more and more crucial to un-derstand their decision-making process. Traditional explainability meth-ods, designed for classification tasks, struggle with modern autoregressiveVLMs due to their complex token-by-token generation process and in-tricate interactions between visual and textual modalities. We presentDEX-AR (Dynamic Explainability for AutoRegressive models), a novelexplainability method designed to address these challenges by generat-ing both per-token and sequence-level 2D heatmaps highlighting imageregions crucial for the model’s textual responses. The proposed methodoffers to interpret autoregressive VLMs, accounting for the varying im-portance of layers and generated tokens—by computing layer-wise gradi-ents with respect to attention maps during the token-by-token generationprocess. DEX-AR introduces two key innovations: a dynamic head filter-ing mechanism that identifies attention heads focused on visual informa-tion, and a sequence-level filtering approach that aggregates per-tokenexplanations while distinguishing between visually-grounded and purelylinguistic tokens. Our evaluation on ImageNet, VQAv2, and PascalVOC,shows a consistent improvement in both perturbation-based metrics, us-ing a novel normalized perplexity measure, as well as segmentation-basedmetrics. Code is available at https://walidbousselham.com/DEX-AR