Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs

Vision-Language Models (VLMs) have demonstrated excep-tional performance in various multi-modal tasks. Recently, there hasbeen an increasing interest in improving the personalization capabilitiesof VLMs. To better integrate user-provided concepts into VLMs, manymethods use positive and negative samples to fine-tune these models.However, the scarcity of user-provided positive samples and the low qual-ity of retrieved negative samples pose challenges for existing techniques.To reveal the relationship between sample and model performance, wesystematically investigate the amount and diversity impact of positiveand negative samples (easy and hard) on VLM personalization tasks.Based on the detailed analysis, we introduce Concept-as-Tree (CaT),which represents a concept as a tree structure, thereby enabling the datageneration of positive and negative samples with varying difficulty anddiversity, and can be easily extended to multi-concept scenarios. Witha well-designed data filtering strategy, our CaT framework can ensurethe quality of generated data, constituting a powerful pipeline. We per-form thorough experiments with various VLM personalization baselinesto assess the effectiveness of the pipeline, alleviating the lack of positivesamples and the low quality of negative samples. Our results demon-strate that CaT equipped with the proposed data filter significantlyenhances the capabilities of VLMs across personalization benchmarks.To the best of our knowledge, this work is the first controllable syn-thetic data pipeline for VLM personalization. The code is released athttps://github.com/zengkaiya/CaT.