EmbedCopilot: Evaluating Vision-Language Models for Hardware-Aware Embedded System Development

We introduce EmbedCopilot-Bench, a multimodal bench-mark designed to evaluate large vision-language models (LVLMs) as as-sistants for embedded development. Compared with pure software pro-gramming, embedded programming inherently involves hardware, mak-ing it well-suited to benefit from LVLMs’ visual capabilities. However, tothe best of our knowledge, no existing benchmark comprehensively eval-uates how effectively LVLMs can serve as embedded copilots. Our bench-mark is developed to fill this gap. Built on real-world embedded develop-ment videos, we constructed 216 annotated multimodal QA triplets span-ning heterogeneous hardware platforms and peripheral modules, cov-ering hardware operation, software configuration, and code generationtasks. To assess model performance, we pair a rubric-driven LLM-as-a-Judge protocol with execution-centric evaluation in Wokwi platformand hardware-in-the-loop Execution Success Rate (ESR), jointly cap-turing semantic quality and functional correctness. Experiments on arange of closed- and open-source LVLMs show that visual context sig-nificantly boosts performance, especially for hardware-related tasks, butthat models remain far from reliable embedded copilots. ESR analysesand a pin-localization case study reveal frequent near-miss failures causedby brittle grounding and limited structured reasoning over board layouts.EmbedCopilot-Bench provides a challenging, realistic testbed for futuremethods that more tightly integrate perception, circuit understanding,and executable code synthesis for real-world embedded development.