Re-test finds defects in all eleven LLM C-to-Rust translators examined

A preprint re-ran the released Rust from eleven AI translation tools against the original C programs. Every tool shipped faulty code, and most faults sat on inputs the tools' own checks never tried.

5 min readIIIOpen sourcePreprint, not peer reviewed
Save
A still life photograph shows a wire sieve containing pale seeds resting offset over a wooden bowl, on warm off-white paper, where a single blue seed sits below.AI illustration
Illustration: AI generated

What the study shows

Rust is a programming language that prevents many of the memory errors behind security flaws in C. Many research systems now use large language models (LLMs) to translate C into Rust automatically. Each reports a success rate based on its own correctness check: a fuzzer (which throws many generated inputs at the code), a model checker, symbolic execution or a test suite.

The author surveyed 34 such translators. Only six report results on inputs their repair loop never saw, the loop being where the LLM revises its Rust until checks pass. Only one, Heimdall, tests its check by feeding it translations known to be wrong. Eleven release both the original C and their Rust in a form that can be run.

The author re-tested all eleven. Fixed scripts generated inputs, and both versions ran side by side with every output compared. A difference counted only where the C was well defined, meaning it passed two standard memory and undefined-behaviour detectors under two compilers.

The author reports 160 defects across all eleven systems. Of these, 93 are outright errors: wrong outputs or crashes on ordinary inputs. The systems' own checks had accepted 86 defective translations, in seven systems. For 81 of those, the check never ran an input that exposed the fault.

The examples are concrete. In one translated copy of the Unix tool tail, the option for 'last 3 bytes' printed the last 3 lines instead, yet its log recorded a pass. A model checker in another system could never reach the assertion it was meant to test. A third check read only a test's exit code, and that test exited cleanly even when it printed a failure.

Where a check did compare outputs fully on the right inputs, it worked. ACTOR's own inputs exposed defects only in translations it had rejected. This is a preprint by a single author and has not been peer reviewed.

Our reading

In our reading, the central finding is not that LLMs write buggy Rust. That was expected. It is that the success rates in this literature mostly measure how thorough each check was, and few papers say. A tool can report a high pass rate simply because its check rarely looks where the bugs are.

The failures are mostly ordinary. Number parsing that rejects a leading space. A length argument that is ignored. Bytes added instead of packed. These are the kinds of slip a careful human reviewer would catch with a handful of edge-case inputs, and they would matter in production code.

The author's four recommendations are cheap: compare every output on inputs the repair loop never saw, confirm a verifier actually reaches its assertion, release exactly what the check accepted, and report how the check itself was tested. None needs new research.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetThe work is a research study with a defined method, run in Docker images built from digest-pinned bases.
II  BenchmarkedMetIt is benchmarked against the artifacts of eleven published systems, with mutation analysis to measure harness power (20% to 91%). It is not a public leaderboard, so this is a research evaluation.
III  Open sourceMetThe paper states the artifact, with all files and results, is at github.com/muhammad-hassnain/retest-c-to-rust. We could not inspect the repository or its licence from the paper.
IV  In productionNot yetThe paper documents no adoption by a named organisation or use of the method in a production translation pipeline.

What would move it up
Independent replication, or adoption of the re-test method by the tested systems' authors or a named organisation, would support level 4. Confirmation that the artifact has a clear reuse licence and runs on other systems would firm up level 3. Peer review would add weight.

The study

Muhammad Hassnain. Re-Testing LLM-Based C-to-Rust Translations Against Their Original C. arXiv, 18 November 2026. Read the paper

Preprint, not peer reviewed · Licence arXiv (no reuse)

Notes

  1. Hassnain, M. Re-Testing LLM-Based C-to-Rust Translations Against Their Original C. arXiv preprint, 18 November 2026, not peer reviewed. Source
  2. Re-test artifact with files and results, as stated in the paper. Repository contents and licence not checked by us. Source
  3. Exact figures: 160 defects in 126 translations; 93 outright errors, 54 edge-input defects; 86 defective translations accepted in seven systems, 56 with an outright error; 81 of 86 never run on an exposing input; harness power 20% (C2SaferRust) to 91% (PtrTrans).
  4. Systems re-tested: VERT, FLOURINE, Syzygy, C2SaferRust, SACTOR, RustMap, RustAssure, PtrTrans, His2Trans, AdaTrans, ACTOR.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Code