Show Me Examples: Inferring Visual Concepts from Image Sets

Vision-language models (VLMs) can follow complex textualinstructions, yet they struggle to reason from purely visual context. Inparticular, current models fail to infer shared concepts from sets of exam-ple images and apply them to new inputs. We introduce Visual ConceptInference from Sets (VICIS), a task that evaluates this capability. Givena small context set of images sharing a concept and a query image, themodel must generate new images that preserve the context-defined conceptwhile remaining consistent with the query. We show that state-of-the-artVLMs perform poorly on this task, often ignoring the visual contextor defaulting to biased generations. To address this gap, we propose atraining framework and architecture that learn to infer visual conceptsfrom image sets and extract concept-specific embeddings from queries.Experiments on synthetic data and large-scale ImageNet/WordNet datashow that our model generates more accurate and diverse outputs andgeneralizes to unseen concepts and modalities such as sketches.