Ceptor: Vision-Language Model-Infused Diverse Guidance for Detecting Anything

Open-set object detection leverages language prompts toguide the perception of categories outside the training set. However,for categories that are difficult to describe or cannot be effectively ab-stracted semantically, models exhibit reduced receptiveness to guidanceprovided by textual descriptions. Recent approaches have explored vi-sual prompting, but they often lack inherent generalization capabilitiesand suffer from limitations in semantic, spatial, and contextual aware-ness during prompt construction. Additionally, isolated prompt path-ways hinder unified optimization. Inspired by the general process of hu-man object search, we designed Ceptor, a unified detector guided bydiverse prompts for open-set object detection. Specifically, we proposeInfusion Strategies to infuse our framework with the generalizationcapabilities of a vision-language model (VLM), leveraging its alignmentproperties and generalizable feature space to construct diverse promptsfor different object types and varying category distributions. Further-more, we introduce a novel prompt generation method, PRE, whicheffectively preserves generalization, contextual information, and spatialaccuracy. Harmonization strategies are also incorporated to ensurecoordinated optimization across various prompts. Ceptor achieves strongperformance with various prompts across multiple datasets, demonstrat-ing the effectiveness of our methodology. Models and code are releasedat https://github.com/jinyanglii/Ceptor.