UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Large Vision-Language Models (LVLMs) remain bottleneckedby massive computational footprints, precluding their deployment onresource-constrained edge devices. While efforts to compress LVLMs fo-cus heavily on vision token reduction or smaller language models, thevision encoder is largely overlooked, typically deployed as a monolithic,computationally heavy feature extractor. Moreover, there is no previouseffort that designs a vision encoder for LVLMs directly optimized foron-device latency. In this paper, we present UltraViT, a vision encoderfor LVLMs, explicitly designed and optimized for on-device performance.Specifically, by taking into account real on-device latencies, we system-atically design a pyramidal architecture that strategically integrates andadapts heterogeneous spatial mixers at the macro-block level. Further-more, to pre-train UltraViT, we propose a novel two-stage generativepre-training strategy: cultivating rich spatial features via dense distil-lation, followed by direct generative supervision from a capacity-mixedfrozen LLM. Compared to standard contrastive and SSL, we show thatour pre-training is much more effective for achieving high-level seman-tic grounding for UltraViT needed for the subsequent generative multi-modal alignment of LVLM training. Extensive experiments demonstratethat our on-device latency-informed design combined with our tailoredtraining strategy establishes a new state-of-the-art for efficient LVLMencoding, significantly outperforming existing encoder-centric baselineswhile operating on-device at nearly 1.7× the speed.