# Correct AI-written code can still run more than three times slower

A new open leaderboard, PerfArena, times the correct code of 13 language models in seven programming languages. Passing the tests turns out to say little about how fast or energy-hungry the code is.

- Category: Code
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-16
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/correct-ai-written-code-can-still-run-more-than-three-times-slower

## The study

PerfArena: Ranking LLMs by Code Performance (preprint, not peer reviewed). arXiv 2611.18030. https://arxiv.org/abs/2611.18030

## At a glance

- What it shows: Thirteen AI models write code that is almost always correct. Some models' correct code still takes noticeably longer to run and uses more energy. On heavy tasks it can take more than three times as long. One model, gpt-6-astra, was never clearly beaten on speed, energy or memory.
- What it does not show: It does not show how this code compares with code written by people. It does not show whether the ranking holds on other computers or on real company codebases.
- Why it matters: Firms now ship a large share of AI-written code, and slow code costs money and electricity every time it runs. Choosing a model only on pass rate or price can miss that cost.

## What the study shows

The authors built PerfArena, a public leaderboard for AI code generators. Most coding benchmarks only ask whether code passes its tests. PerfArena first checks correctness. It then measures three things about every correct program: latency (how long it takes to run), peak memory, and energy use.

They tested 13 large language models (LLMs, the AI systems behind coding assistants) from six providers. The models wrote code in seven languages, including Python, Java, C++ and TypeScript. There were two task sets. One holds 99 LeetCode puzzles, chosen because fast and slow solutions differ clearly. The other holds 10 heavy programs from a well-known cross-language speed test, the Computer Language Benchmarks Game. Each model gave one answer per task and language, from the same prompt.

Correctness barely separated the models on the puzzles: 12 of 13 passed more than 90%. Speed did separate them. The authors report that the slowest model's correct code took about 1.6 times as long to run as the leader's. On the heavier programs, the gap reached about 3.6 times.

To rank models, the authors compare two models' solutions to the same task in the same language, a 'duel'. They count the wins, and differences smaller than measurement noise count as ties. Across both task sets, gpt-6-astra was the only model that no other model beat significantly on any of the three measures. On the puzzles, claude-opus-5-5 tied with it for fastest code.

The best model depends on the language. On the puzzles, claude-opus-5-5 led in Python and PHP, and gpt-6-astra led in the other five languages. Energy use tracked speed closely. Peak memory hardly separated the models on the puzzles.

Price was a poor guide to speed. The most expensive model per correct solution cost about 2.6 times as much as gpt-6-astra on the puzzles, without producing faster code. The cheapest models cost 8 to 12 times less, but their code ran slower. The paper is a preprint and has not been peer reviewed.

## Our reading

In our reading, the main contribution is the measuring method, more than any single winner. Each comparison holds the task, the language, the inputs and the machine fixed. Only correct programs are timed, and every timing is repeated seven times. That makes the efficiency claims more careful than most earlier code benchmarks.

The practical point is simple. A pass rate does not show what the code will cost to run. Consider a team that uses an assistant to write a data-processing job in C# that runs millions of times a day. On these benchmarks, two models with similar pass rates could produce code whose running time differs by a third or more. That gap could mean more servers and a higher power bill. The authors' own advice is to compare models per language and per measure.

Readiness: III, open source available. The authors report public benchmark results and say the code is released under an open licence. They name no organisation using the rankings in production. We have not inspected the repository or the live leaderboard.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/correct-ai-written-code-can-still-run-more-than-three-times-slower

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
