Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners

Multimodal Large Language Models (MLLMs) have demon-strated remarkable abilities when analyzing images, yet translating thesecapabilities to few-shot image classification remains challenging. To bridgethis gap, we present DeCoDe, a simple yet effective technique that en-ables off-the-shelf MLLMs to act as strong few-shot classifiers withoutany additional training. Our approach builds on the idea of few-shotclassification as a set of pairwise image comparisons, decomposing thetask into a set of binary decisions. Given a query image and a supportimage from a candidate class, the MLLM is prompted to decide whetherthe two images depict the same class. The logit corresponding to an af-firmative response is then used as a similarity score to assign the queryimage to the most likely class. While this already yields good results,we show that providing additional high-level information, such as thedata domain, to the model further improves performance. Our evalu-ation provides an extensive analysis of various inference variants on asuite of twelve datasets, six established and six newly curated few-shotbenchmarks spanning across diverse domains. The results show that theproposed simple decomposition technique can turn off-the-shelf MLLMsinto powerful few-shot learners, significantly outperforming current state-of-the-art few-shot methods on both standard and novel domains. Codeis available at https://github.com/yunhanwang1105/DeCoDe.