Correct AI-written code can still run more than three times slower

A new open leaderboard, PerfArena, times the correct code of 13 language models in seven programming languages. Passing the tests turns out to say little about how fast or energy-hungry the code is.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
This is the cover image for an article on applied AI research in *Science Underway*. Let me know if this aligns with what you had in mind.AI illustration
Illustration: AI generated

What the study shows

The authors built PerfArena, a public leaderboard for AI code generators. Most coding benchmarks only ask whether code passes its tests. PerfArena first checks correctness. It then measures three things about every correct program: latency (how long it takes to run), peak memory, and energy use.

They tested 13 large language models (LLMs, the AI systems behind coding assistants) from six providers. The models wrote code in seven languages, including Python, Java, C++ and TypeScript. There were two task sets. One holds 99 LeetCode puzzles, chosen because fast and slow solutions differ clearly. The other holds 10 heavy programs from a well-known cross-language speed test, the Computer Language Benchmarks Game. Each model gave one answer per task and language, from the same prompt.

Correctness barely separated the models on the puzzles: 12 of 13 passed more than 90%. Speed did separate them. The authors report that the slowest model's correct code took about 1.6 times as long to run as the leader's. On the heavier programs, the gap reached about 3.6 times.

To rank models, the authors compare two models' solutions to the same task in the same language, a 'duel'. They count the wins, and differences smaller than measurement noise count as ties. Across both task sets, gpt-6-astra was the only model that no other model beat significantly on any of the three measures. On the puzzles, claude-opus-5-5 tied with it for fastest code.

The best model depends on the language. On the puzzles, claude-opus-5-5 led in Python and PHP, and gpt-6-astra led in the other five languages. Energy use tracked speed closely. Peak memory hardly separated the models on the puzzles.

Price was a poor guide to speed. The most expensive model per correct solution cost about 2.6 times as much as gpt-6-astra on the puzzles, without producing faster code. The cheapest models cost 8 to 12 times less, but their code ran slower. The paper is a preprint and has not been peer reviewed.

Our reading

In our reading, the main contribution is the measuring method, more than any single winner. Each comparison holds the task, the language, the inputs and the machine fixed. Only correct programs are timed, and every timing is repeated seven times. That makes the efficiency claims more careful than most earlier code benchmarks.

The practical point is simple. A pass rate does not show what the code will cost to run. Consider a team that uses an assistant to write a data-processing job in C# that runs millions of times a day. On these benchmarks, two models with similar pass rates could produce code whose running time differs by a third or more. That gap could mean more servers and a higher power bill. The authors' own advice is to compare models per language and per measure.

Readiness: III, open source available. The authors report public benchmark results and say the code is released under an open licence. They name no organisation using the rankings in production. We have not inspected the repository or the live leaderboard.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetThe system was built and tested in a lab setting on one bare-metal host with a controlled campaign.
II  BenchmarkedMetIt reports results for 13 LLMs on two public task sets (CLBG programs and LeetCode problems), with significance tests and robustness checks. The authors' own PerfArena leaderboard at perfarena.net is the benchmark.
III  Open sourceMetSource code is released under Apache 2.0 at https://github.com/perfarena/perfarena. Generated solutions, measurements and rankings are released under CC BY 4.0. The paper states this, but we did not inspect the repository.
IV  In productionNot yetThe paper names no organisation using PerfArena in production. It is a benchmark rather than a deployed product, and only the authors submit results.

What would move it up
To reach level 4, a named organisation would need to document using PerfArena rankings to pick or monitor LLMs in real engineering work. Independent replication on other hardware, results with multiple samples per problem, and comparison with human-written code would also strengthen the case.

The study

Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes et al.. PerfArena: Ranking LLMs by Code Performance. arXiv, 13 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY 4.0

Notes

  1. Rua, Mersha, Fernandes and Ali, 'PerfArena: Ranking LLMs by Code Performance', arXiv preprint, 13 November 2026, CC BY 4.0. Not peer reviewed. Source
  2. PerfArena source code, Apache License 2.0, per the paper. We did not inspect it. Source
  3. Live PerfArena leaderboard with downloadable solutions and measurements, per the paper. Source
  4. Exact figures: on LeetCodeComp, the other models' correct code takes 1.14 to 1.61 times as long as gpt-6-astra's. On CLBGComp the range is 1.20 to 3.61 times. pass@1 for gpt-6-astra is 99.7% on LeetCodeComp and 100% on CLBGComp. Latency win probabilities for gpt-6-astra are 62% and 72% (Table 2).
  5. Cost per correct solution for gpt-6-astra is $0.011 on LeetCodeComp and $0.039 on CLBGComp. claude-fable-5-1 costs 2.6 times as much on LeetCodeComp and 5.1 times on CLBGComp. On CLBGComp, the most expensive model per correct solution is glm-5.3-flash, at 9.3 times (Table 3).
  6. Rank correlation between energy and latency rankings: 0.91 on LeetCodeComp and 0.97 on CLBGComp. Mean Kendall's tau between language rankings on LeetCodeComp is 0.50, against 0.72 after shuffling.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Code