Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives

Multimodal large language models (LLMs) are increasinglyused to generate dermatology diagnostic narratives directly from images.However, reliable evaluation remains the primary bottleneck for respon-sible clinical deployment. We introduce a novel evaluation frameworkthat combines DermBench, a meticulously curated benchmark, withDermEval, a robust automatic evaluator, to enable clinically meaning-ful, reproducible, and scalable assessment. We build DermBench, whichpairs 4000 real-world dermatology images with expert-certified diagnos-tic narratives and uses an LLM-based judge to score candidate narrativesacross clinically grounded dimensions, enabling consistent and compre-hensive evaluation of multimodal models. For individual case assessment,we train DermEval, a reference-free multimodal evaluator. Given an im-age and a generated narrative, DermEval produces a structured critiquealong with an overall score and per-dimension ratings. This capabil-ity enables fine-grained, per-case analysis, which is critical for identi-fying model limitations and biases. Experiments on a diverse dataset of4500 cases demonstrate that DermBench and DermEval achieve closealignment with expert ratings, with mean deviations of 0.251 and 0.117(out of 5) respectively, providing reliable measurement of diagnostic abil-ity and trustworthiness across different multimodal LLMs. The bench-mark resources are publicly available at https://github.com/yuhos16/DermBench.