GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

Recent advancements in vision-language models (VLMs) haveleveraged large language models (LLMs) to achieve performance on parwith closed-source systems like GPT-4V. However, deploying these mod-els in real-world scenarios, particularly on resource-constrained devices,remains challenging due to their substantial computational demands.This has spurred interest in distilling knowledge from large VLMs intosmaller, more efficient counterparts. A key challenge arises here from thediversity of VLM architectures, which are built on different LLMs andemploy varying token types—differing in vocabulary size, token splits,Changing Teacher VLMs for Distillation MMMU70GenRecalMMB GPT-4oInternVL2.5-78B → InternVL2.5-8B60 Qwen2-VLAccuracy (%)MM-Vet InternVL2.550 MolmoMMMU40NVLM-72B → InternVL2.5-8B LLaVA-OneVisionInternVL2-76B → InternVL2.5-8BMMMU-Pro Qwen2-VL-72B → InternVL2.5-8BInternVL2.5-78B → InternVL2.5-8B300 20 40 60 80 1 2 4 7 8 … 72 78Accuracy (%) Model Size (B)Fig. 2: (Left) Comparison of the challenging benchmark performances, MMB [36], MM-Vet [63], MMMU [65], and MMMU-Pro [66] by changing large VLMs. The more pow-erful large VLMs we select, the greater the performance improvement we can achieve.(Right) Comparing the performance of the challenging benchmark: MMMU [65], withGenRecal and various VLMs across model sizes. Note that all the experiments in Fig. 1and Fig. 2 are conducted on the equal training dataset.and token index ordering. To address this challenge of limitation to aspecific VLM type, we present Generation after Recalibration (Gen-Recal), a general-purpose distillation framework for VLMs. GenRecalincorporates a Recalibrator that aligns and adapts feature representa-tions between heterogeneous VLMs, enabling effective knowledge trans-fer across different types of VLMs. Through extensive experiments onmultiple challenging benchmarks, we demonstrate that GenRecal signif-icantly improves baseline performances, eventually outperforming large-scale open- and closed-source VLMs.