Coding agents catch more firmware bugs when they can poke virtual hardware

A Duke University team built a simulated workbench where AI coding agents can change sensor readings and press virtual buttons while testing code for small chips. With it, agents passed 89% of tasks, against about 80% with compiler feedback alone.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
A small cardboard model with wooden switches and a brass dial, with a brass rod pressing its single blue button.AI illustration
Illustration: AI generated

What the study shows

Firmware is the software that runs on a microcontroller, the small chip inside devices such as thermostats, locks and sensors. AI coding agents do well on ordinary software because they can run the code and its tests, then fix what fails. Firmware breaks that loop. The code runs on a separate chip and talks to real sensors, so an agent at a text terminal cannot see what goes wrong.

The authors built EmbedGym to close the loop. It uses emulators, programs that imitate a chip instruction by instruction, so the agent's exact compiled program runs as it would on a real board. The emulated chip is wired to 41 simulated parts, such as sensors, displays and motors. It also connects to a circuit simulator and to virtual WiFi and Bluetooth devices. Agents control it with three text commands: set a physical value, do an action such as a button press, and read a component's state.

On top of this they built EmbedGym-Bench, 598 tasks across four chips and three software frameworks. Each task is graded by hidden, timed checks on the simulated hardware. A thermostat-style test might set the temperature, press a button, then check what the display shows. The tasks were largely generated by a pipeline of AI agents, with short human checks at the start and the end.

The authors ran eight AI models at three levels of feedback. At the first, agents saw only compiler messages. At the second, they could also run the program and read its text output, known as serial logs. At the third, they could also change inputs and inspect component states. With compiler feedback alone, almost every program built, yet about one in five failed the hardware checks.

Passive serial logs barely helped. They added under one percentage point overall, because agents broke nearly as many working programs as they fixed. Full interaction raised the overall pass rate to 89.1%. Every model improved, by between about 5 and 15 points.

The authors also report that a cheap model with full interaction scored about the same as two dearer models given compiler feedback only, at 30 to 70 times lower cost per task. On 60 programs also run on physical boards, the simulator and the real hardware agreed on 59. The paper is a preprint and has not been peer reviewed.

Our reading

In our reading, the main lesson is not about which model is best. It is about the test environment. A device sitting on a desk shows constant readings, and constant readings hide bugs. The paper's own example is a light sensor meant to switch into an ultraviolet mode. The buggy code never switched, but indoors the ultraviolet reading is zero anyway. The code looked fine until the agent injected a non-zero ultraviolet value and saw it still read zero.

The same pattern applies widely. Picture a heating controller meant to switch on below 18°C. Tested at room temperature, the faulty threshold never fires, and the logs look clean. An agent that can set the temperature to 15°C finds the bug in one run. The authors' breakdown supports this: the largest gains came on analogue sensors, buttons and threshold logic, where idle values mask errors.

On our four-level readiness scale, we place this at level III, open source available. The work is benchmarked, and the authors say the environment, benchmark, workflow and agent runs are released. We have not checked the repository or its licence. There is no documented use by an outside organisation, so it does not reach level IV.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetA working system exists, with emulators, 41 peripheral models, a CLI and a grading harness. It was also tested on a small set of physical boards.
II  BenchmarkedMetEmbedGym-Bench has 598 tasks. Eight models were evaluated on it and compared with two prior frameworks (EmbedAgent, AutoEmbed). Caveat: the benchmark is the authors' own and new, not an established public benchmark. The result is a preprint.
III  Open sourceMetThe authors say EmbedGym, EmbedGym-Bench, the scaling workflow and all trajectories are open-sourced at https://embedgym.github.io. We could not check the repository, so the licence and completeness are unverified. The paper does not state them.
IV  In productionNot yetThe paper documents no use in production or by any named organisation outside the authors' own experiments.

What would move it up
To reach level 4: documented use by a named team building real firmware, with before-and-after data on bugs caught or time saved. Independent replication on new chips, sensors and tasks, plus a larger real-board validation, would also strengthen the case.

The study

Minghui Zhao, Yiming Li, Sophie Zhu et al.. EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware. arXiv, 16 November 2026. Read the paper

Preprint, not peer reviewed · Licence arXiv (no reuse)

Notes

  1. Zhao et al., EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware. arXiv preprint, 16 November 2026, not peer reviewed. Source
  2. Project site, where the authors say EmbedGym, EmbedGym-Bench, the scaling workflow and all agent runs are released. We have not verified the contents or licence. Source
  3. Exact figures: compiler-only feedback built 99.7% of tasks, with 19.9% of built programs failing hardware checks (up to 73% on single targets). Passive logs moved the overall pass rate from 79.9% to 80.8%, fixing 7.1% and breaking 6.1%. Full interaction reached 89.1%, +4.9 to +14.7 points per model and up to +31 on one target. Gains on ESP-IDF were +11.6 and on Zephyr +14.5. Total runs: 14,352.
  4. Cost comparison: GPT-6 Luna with full interaction scored 89.0% at $0.0026 per task. Grok 4.6 scored 89.5% at $0.075 and Gemini 3.8 Flash 90.3% at $0.19, both with compiler feedback only. Prices as of October 2026, per the paper.
  5. Baselines on Arduino targets: full interaction 90 to 99%, EmbedAgent Build-Fix 77 to 90%, AutoEmbed 67 to 82%. Compile-only and serial self-check criteria accepted 25 to 39% of firmwares that failed the hardware checks.
  6. Real-board check: 59 of 60 programs agreed (98.3%). The mismatch involved an nRF52840 compass task and an HMC5883L-compatible clone chip that did not enforce a register lock described in the datasheet.
  7. Readiness: level III, open source available, based on the authors' statement of release. Not level IV: no documented production use by a named organisation.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Code