Delineating Knowledge Boundaries for Honest Large Vision-Language Models

Large Vision-Language Models (VLMs) have achieved re-markable multimodal performance yet remain prone to factual halluci-nations, particularly in long-tail or specialized domains. Moreover, cur-rent models exhibit a weak capacity to refuse queries that exceed theirparametric knowledge. In this paper, we propose a systematic frame-work to enhance the refusal capability of VLMs when facing such un-known questions. We first curate a model-specific "Visual-Idk" (Visual-Idon’t know) dataset, leveraging multi-sample consistency probing to dis-tinguish between known and unknown facts. We then align the modelusing supervised fine-tuning followed by preference-aware optimization(e.g., DPO, ORPO) to effectively delineate its knowledge boundaries. Re-sults on the Visual-Idk dataset show our method improves the TruthfulRate from 57.9% to 67.3%. Additionally, internal probing also demon-strates that the model genuinely recognizes its boundaries instead of justmemorizing refusal patterns. Our framework further generalizes to out-of-distribution medical and perceptual domains, providing a robust pathtoward more trustworthy and prudent visual assistants.