Claude Code agents built post-quantum FPGA hardware, but their tests missed faults
A preprint follows AI coding agents as they port a quantum-safe encryption chip design to new hardware. The hardware works, but the agents' own tests and reviews proved weaker than they looked.

AI illustrationWhat the study shows
The authors, from Lucid Motors and EYL Inc., used AI agents in the Claude Code harness to port a hardware accelerator. The accelerator runs ML-KEM and ML-DSA, the new US standards for quantum-safe key exchange and digital signatures. It went from an older FPGA card (a reprogrammable chip) to a newer Zynq UltraScale+ card running three times faster, at 300 MHz. This is a preprint and has not been peer reviewed.
The agents worked in a seven-stage workflow. Every claim about the design became an automatic check, called a gate. A gate only counted if it was shown to fail on deliberately broken code. Separate reviewer agents then audited each piece before it was saved.
The port worked. All six operations gave byte-exact results on the board. It passed all 85 cases of NIST's official conformance tests that the device's interface could accept, out of 285 in total. It also passed 484 extra signature checks.
The authors then tested the tests. They used a standard tool to make hundreds of copies of three modules, each with one random bit of the circuit changed. These broken copies are called mutants. The existing tests caught 76% of them. For the key-exchange decryption module, the figure was only about half.
An agent then wrote five new tests from the specification, guided by the mutants that slipped through. On a fresh batch of mutants, these raised the decryption module's catch rate from 48% to 74%.
The authors also replayed AI code review on four known flaws. These were flaws that standard known-answer tests, which only check correct outputs for valid inputs, cannot see. When reviewers were pointed at the flawed part, they found every flaw in all 54 sessions. When asked to review the whole design, they often missed two flaws in the signature checker, finding one of them in only 1 of 6 sessions.
Finally, the authors planted small one-line flaws in the key-protection logic. Reviewers pointed at that part found them in only 3 of 20 sessions. Formal checks, which mathematically prove a property holds, flagged every one.
Our reading
In our reading, the headline is not that agents can build cryptographic hardware. It is that the evidence agents produce about their own work is softer than it appears. Every test passed, every review approved, and yet a quarter of random faults went unnoticed. A product team reading a green dashboard would have seen nothing wrong.
The review results point to a practical rule. AI reviewers did well when told exactly where to look, and poorly when handed a whole design. That matters because, in real projects, nobody knows in advance where the flaw is. The study suggests keeping review narrow, one module at a time, and before every commit.
For the most sensitive logic, the part that keeps secret keys inside the device, neither tests nor AI review were enough. Only formal checks caught the planted flaws. On our Readiness Scale we rate this work at level II, Benchmarked: it reports results against NIST's public conformance vectors, but no code or designs are released.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | Working research prototype on real hardware, with a described workflow and an evaluation. |
| II Benchmarked | Met | Results against standard NIST ACVP vectors (85 applicable cases, all passed) and known answers on the board. There are also measured mutant kill rates and replay results. There is no shared public leaderboard benchmark, and the ACVP coverage is partial (85 of 285), so this is a modest pass. |
| III Open source | Not yet | The paper links no code, weights or data for this work. It says the base protocol of [6] is published as a repository template, but gives no URL here, and the B port is not stated to be released. |
| IV In production | Not yet | No documented production use by a named organisation. The work is a research port on one development card, with a prototype key scheme tied to a readable device ID. |
What would move it up
Releasing the workflow, gates, mutant scripts, replay prompts and B's RTL under an open licence would reach level 3. Level 4 would need a named organisation using a hardened version in a product. That would need a full ACVP pass, side-channel review and independent replication with other models and a human baseline.
The study
Jungmin Park, Eunha Kim, Wooseop Kim et al.. From FIPS to FPGA with LLM Agents: Measuring What Tests and Reviews Miss in Post-Quantum Cryptographic Hardware. arXiv, 19 November 2026. Read the paper
Notes
- Park et al., From FIPS to FPGA with LLM Agents: Measuring What Tests and Reviews Miss in Post-Quantum Cryptographic Hardware. arXiv preprint, 19 November 2026, not peer reviewed. Source
- No code, designs or data are linked in the paper. The earlier protocol by the same authors is described as a repository template, but no link is given.
- Exact mutant figures: 303 of 399 first-draw mutants killed (76%); per module 92% transform, 67% verifier, 53% decapsulation. Of 96 survivors, 19 provably equivalent, 22 provably different, 55 undecided, giving 80 to 93% on behaviour-changing mutants. Fresh draw of 150: 70.7% rising to 81.3% with new tests.
- Review replay figures: 54 of 54 scoped sessions found each known flaw; whole-design plain request found missing hint checks in 3 of 6 and unchecked z coefficients in 1 of 6; planted custody flaws found in 3 of 20, planted verifier flaws in 20 of 20; false alarms on clean code in 3 of 24.
- Mutants were made with Yosys mutate, the command behind the open-source MCY tool. Formal checks used SymbiYosys with ABC and rIC3. Source
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

