Unleashing the Power of Large-Scale ViT in Zero-Shot SBIR: A Strong Baseline with Multi-Layer Feature Aggregation

The domain gap between abstract sketches and natural im-ages, combined with the challenge of zero-shot generalization, remainsa critical bottleneck in Sketch-Based Image Retrieval (ZS-SBIR). Whilescaling up to large-scale Vision-Language Models offers powerful baselinerepresentations, naive utilization—such as relying solely on the terminal[CLS] token—fails to fully exploit their potential, often discarding thefine-grained structural primitives essential for abstract sketch alignment.In this paper, we propose a robust framework that bridges this modalitygap by decoupling and composing visual states and semantics. First, ourMulti-Layer Feature Aggregation (MLFA) acts as an anti-compressionmechanism, explicitly reviving low-level geometric primitives alongsidehigh-level semantic objects. Second, a Phased Cross-Modal Interaction(PCMI) serves as a dynamic compositional engine: learnable promptsprogressively query and bind these structural states with global seman-tics for precise coarse-to-fine alignment. Extensive experiments demon-strate our framework achieves state-of-the-art results on several bench-marks and remains competitive on QuickDraw Ext.