# Coding agents catch more firmware bugs when they can poke virtual hardware

A Duke University team built a simulated workbench where AI coding agents can change sensor readings and press virtual buttons while testing code for small chips. With it, agents passed 89% of tasks, against about 80% with compiler feedback alone.

- Category: Code
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-18
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/coding-agents-catch-more-firmware-bugs-when-they-can-poke-virtual-hardware

## The study

EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware (preprint, not peer reviewed). arXiv 2611.21845. https://arxiv.org/abs/2611.21845

## At a glance

- What it shows: Code for the small chips inside devices often builds without errors but still misbehaves when it runs. Letting AI coding agents test that code on simulated hardware, where they can change sensor readings and press buttons, raised their success rate from about 80% to 89% of tasks.
- What it does not show: It does not show the approach works for real products at scale. Only 60 programs were checked on physical boards, and no company has reported using it.
- Why it matters: Bugs in device code reach the physical world, from sensors to locks. The study suggests that checking only whether such code builds lets a large share of broken programs through.

## What the study shows

Firmware is the software that runs on a microcontroller, the small chip inside devices such as thermostats, locks and sensors. AI coding agents do well on ordinary software because they can run the code and its tests, then fix what fails. Firmware breaks that loop. The code runs on a separate chip and talks to real sensors, so an agent at a text terminal cannot see what goes wrong.

The authors built EmbedGym to close the loop. It uses emulators, programs that imitate a chip instruction by instruction, so the agent's exact compiled program runs as it would on a real board. The emulated chip is wired to 41 simulated parts, such as sensors, displays and motors. It also connects to a circuit simulator and to virtual WiFi and Bluetooth devices. Agents control it with three text commands: set a physical value, do an action such as a button press, and read a component's state.

On top of this they built EmbedGym-Bench, 598 tasks across four chips and three software frameworks. Each task is graded by hidden, timed checks on the simulated hardware. A thermostat-style test might set the temperature, press a button, then check what the display shows. The tasks were largely generated by a pipeline of AI agents, with short human checks at the start and the end.

The authors ran eight AI models at three levels of feedback. At the first, agents saw only compiler messages. At the second, they could also run the program and read its text output, known as serial logs. At the third, they could also change inputs and inspect component states. With compiler feedback alone, almost every program built, yet about one in five failed the hardware checks.

Passive serial logs barely helped. They added under one percentage point overall, because agents broke nearly as many working programs as they fixed. Full interaction raised the overall pass rate to 89.1%. Every model improved, by between about 5 and 15 points.

The authors also report that a cheap model with full interaction scored about the same as two dearer models given compiler feedback only, at 30 to 70 times lower cost per task. On 60 programs also run on physical boards, the simulator and the real hardware agreed on 59. The paper is a preprint and has not been peer reviewed.

## Our reading

In our reading, the main lesson is not about which model is best. It is about the test environment. A device sitting on a desk shows constant readings, and constant readings hide bugs. The paper's own example is a light sensor meant to switch into an ultraviolet mode. The buggy code never switched, but indoors the ultraviolet reading is zero anyway. The code looked fine until the agent injected a non-zero ultraviolet value and saw it still read zero.

The same pattern applies widely. Picture a heating controller meant to switch on below 18°C. Tested at room temperature, the faulty threshold never fires, and the logs look clean. An agent that can set the temperature to 15°C finds the bug in one run. The authors' breakdown supports this: the largest gains came on analogue sensors, buttons and threshold logic, where idle values mask errors.

On our four-level readiness scale, we place this at level III, open source available. The work is benchmarked, and the authors say the environment, benchmark, workflow and agent runs are released. We have not checked the repository or its licence. There is no documented use by an outside organisation, so it does not reach level IV.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/coding-agents-catch-more-firmware-bugs-when-they-can-poke-virtual-hardware

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
