Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition

The incorporation of additional modalities into action recog-nition models increases their performance across a wide range of settings.However, how this additional information can contribute to making themodels more robust remains underexplored, particularly for the case ofmulti-modal out-of-distribution (OOD) detection. While methods existthat regularize the multi-modal training process with OOD detection inmind, they still apply off-the-shelf OOD detectors designed for the uni-modal case during inference, discarding important information. Basedon an interesting relationship we find between the multi-modal and uni-modal predictions, we propose to use this signal to build a post-hocdetector explicitly designed for the multi-modal scenario. We combinethis new source of information with a feature-space score, which detectsoff-manifold samples in the multi-modal space, and normalize them bythe multi-modal logits. In doing so, the proposed hybrid detector is com-patible with existing training-time approaches and consistently improvesperformance. Experiments on a wide range of established datasets fromthe MultiOOD benchmark show that, on average, our approach outper-forms the state of the art. Our results show the importance of explicitlyconsidering the different modalities at inference time for multi-modalOOD detection.