MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

Prompt learning has become a dominant paradigm for adapt-ing vision-language models (VLMs) such as CLIP to downstream taskswithout modifying pretrained weights. While extending prompts to bothvision and text encoders across multiple transformer layers significantlyboosts performance, it dramatically increases the number of trainableparameters, with state-of-the-art methods requiring millions of parame-ters and abandoning the parameter efficiency that makes prompt tuningattractive. In this work, we propose MMLoP (Multi-Modal Low-RankPrompting), a framework that achieves deep multi-modal promptingwith only 11.5K trainable parameters, comparable to early text-onlymethods like CoOp. MMLoP parameterizes vision and text prompts ateach transformer layer through a low-rank factorization that constrainsprompts to a compact subspace, providing parameter efficiency whilemotivating the need for our complementary regularization components.To further close the accuracy gap with state-of-the-art methods, we intro-duce three complementary components: a self-regulating consistency lossthat anchors prompted representations to frozen zero-shot CLIP featuresat both the feature and logit levels, a uniform drift correction that re-moves the global embedding shift induced by prompt tuning to preserveclass-discriminative structure, and a shared up-projection that couplesvision and text prompts through a common low-rank factor to enforcecross-modal alignment. Extensive experiments across three benchmarksand 11 diverse datasets demonstrate that MMLoP achieves a highly favor-able accuracy-efficiency tradeoff, outperforming the majority of existingmethods including those with orders of magnitude more parameters, whileachieving a harmonic mean of 79.70% on base-to-novel generalization.Code is available at https://github.com/sajjad-ucsb/MMLoP.