# Re-test finds defects in all eleven LLM C-to-Rust translators examined

A preprint re-ran the released Rust from eleven AI translation tools against the original C programs. Every tool shipped faulty code, and most faults sat on inputs the tools' own checks never tried.

- Category: Code
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-19
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/re-test-finds-defects-in-all-eleven-llm-c-to-rust-translators-examined

## The study

Re-Testing LLM-Based C-to-Rust Translations Against Their Original C (preprint, not peer reviewed). arXiv 2611.23839. https://arxiv.org/abs/2611.23839

## At a glance

- What it shows: Eleven research tools that use large language models to turn C code into Rust all released translations that behave differently from the original. The tools' own checks had approved 86 of these faulty translations.
- What it does not show: It does not give a corrected success rate for any tool. It also does not say how often AI translation fails in general, because only tools with runnable releases were tested.
- Why it matters: Governments and DARPA are pushing a move from C to memory-safe Rust, and AI translation is the fast route. A reported success rate is only as trustworthy as the check behind it.

## What the study shows

Rust is a programming language that prevents many of the memory errors behind security flaws in C. Many research systems now use large language models (LLMs) to translate C into Rust automatically. Each reports a success rate based on its own correctness check: a fuzzer (which throws many generated inputs at the code), a model checker, symbolic execution or a test suite.

The author surveyed 34 such translators. Only six report results on inputs their repair loop never saw, the loop being where the LLM revises its Rust until checks pass. Only one, Heimdall, tests its check by feeding it translations known to be wrong. Eleven release both the original C and their Rust in a form that can be run.

The author re-tested all eleven. Fixed scripts generated inputs, and both versions ran side by side with every output compared. A difference counted only where the C was well defined, meaning it passed two standard memory and undefined-behaviour detectors under two compilers.

The author reports 160 defects across all eleven systems. Of these, 93 are outright errors: wrong outputs or crashes on ordinary inputs. The systems' own checks had accepted 86 defective translations, in seven systems. For 81 of those, the check never ran an input that exposed the fault.

The examples are concrete. In one translated copy of the Unix tool tail, the option for 'last 3 bytes' printed the last 3 lines instead, yet its log recorded a pass. A model checker in another system could never reach the assertion it was meant to test. A third check read only a test's exit code, and that test exited cleanly even when it printed a failure.

Where a check did compare outputs fully on the right inputs, it worked. ACTOR's own inputs exposed defects only in translations it had rejected. This is a preprint by a single author and has not been peer reviewed.

## Our reading

In our reading, the central finding is not that LLMs write buggy Rust. That was expected. It is that the success rates in this literature mostly measure how thorough each check was, and few papers say. A tool can report a high pass rate simply because its check rarely looks where the bugs are.

The failures are mostly ordinary. Number parsing that rejects a leading space. A length argument that is ignored. Bytes added instead of packed. These are the kinds of slip a careful human reviewer would catch with a handful of edge-case inputs, and they would matter in production code.

The author's four recommendations are cheap: compare every output on inputs the repair loop never saw, confirm a verifier actually reaches its assertion, release exactly what the check accepted, and report how the check itself was tested. None needs new research.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/re-test-finds-defects-in-all-eleven-llm-c-to-rust-translators-examined

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
