SpecV: Specification Verification for Robust Unified Multimodal Evaluation

Unified Multimodal Models (UMMs) consolidate visual un-derstanding, image generation, editing, and interleaved image-text in-teraction, yet their evaluation increasingly depends on VLM-as-a-judgescoring that shifts as judges, prompts, or model versions evolve — aproblem we term evaluation drift. We present SpecV, a specification-verification framework for robust unified multimodal evaluation. SpecVreplaces holistic scoring with the Specification Verification Proto-col, which decomposes each prompt into atomic, binary specificationsand verifies them against model outputs, improving cross-judge agree-ment and stability while preserving alignment with human judgments.To produce reliable specifications, we propose Specification Ensem-ble and Refinement, a multi-model pipeline that aggregates candidatespecifications, deduplicates them semantically, and filters for verifiabil-ity and relevance. We also introduce SpecV-Bench, a 1,200-instancebenchmark covering six core UMM tasks with sub-tracks of increasingconstraint complexity, enabling fine-grained analysis of capability trade-offs and failure modes. Across extensive experiments with multiple judgemodels, SVP consistently reduces ranking flips and improves evalua-tion reproducibility. Benchmark is available at https://github.com/yuyouxixi/SpecV.