Self-improving AI agents stall because noisy tests cannot spot real gains
A large reproducibility study finds that agents which rewrite their own prompts, skills and memories often fail because noisy evaluations cannot tell good changes from bad ones. Many published method rankings may rest on luck.


What the study shows
The paper studies agent evolution. This is a loop where an AI agent writes new versions of its own prompts, skills or memories, tests them, and keeps the ones that score higher. The model itself is not retrained. The authors ran seven published evolution methods with six models on four agent benchmarks, using over 10 million model calls. This is a preprint and has not been peer reviewed.
The authors propose what they call the Resolution Hypothesis. Their claim is that the bottleneck is not writing better versions. It is recognising which versions are actually better, because each test run of an agent gives a slightly different score.
First, they measured how much scores vary. A method run once and tested once has a typical spread of about 2 to 4 percentage points. They surveyed nine published papers and found the winning method usually leads its best rival by a median of 3.4 points. None of those results reported a variance estimate. On Terminal-Bench, a coding benchmark run in a command-line environment, a single run picked the true best of four methods only 33% of the time, against 25% for a random guess.
Second, they looked at the largest reported gains, often 10 to 40 points. The authors report that adding one hand-written sentence to the unimproved agent recovered a median of 90% of the gain. These sentences usually exploited a quirk in the grader, such as writing spreadsheet values instead of formulas, or matching a strict answer format. After the graders were fixed, the median gain fell from about 18 points to under 1.
Third, they tested the loop step by step on Terminal-Bench. A pool of 100 candidate skills did contain genuinely better ones. But the usual selection rules picked the better of two skills about half the time, no better than a coin toss. Asking a language model to judge the agent's work did not help. Over successive rounds, the scores the loop recorded for itself rose by 7 to 9 points, while careful re-testing showed little or no real improvement.
Our reading
In our reading, this is a measurement paper more than a methods paper, and an important one. Its core message is simple. If a test cannot reliably detect a 3 point improvement, then a loop that keeps whatever scores highest will mostly keep lucky draws. The same noise also makes one run of one method a weak basis for ranking methods against each other.
The shortcut audit is the most striking part. It suggests that the large gains which made this field look promising often came from the agent learning what the grader wanted to see, not from becoming more capable. That is a familiar problem in machine learning, but it is easy to miss when the thing being tuned is plain-language instructions.
On our four-level readiness scale, we rate this work II, Benchmarked. It reports results on public benchmarks. The authors say code and data will be released only after acceptance, so it does not yet meet level III.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The work is a research study with controlled experiments, so it meets the lab-only baseline. |
| II Benchmarked | Met | Results are reported on public benchmarks: ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1 and nine others. Seven existing methods and six models were run, with variance estimates. This is a benchmarking and audit study, not a new method. |
| III Open source | Not yet | The paper says reproducible materials and trajectory data will be released only upon acceptance. A project page is named, but the paper cites no released code, weights or data. The methods studied (GEPA, SkillOpt-Lite) have public code, but that is not this paper's contribution. |
| IV In production | Not yet | No documented use by a named organisation in production. The paper is a research diagnostic, and the suggested practices (variance reporting, component-wise evaluation) have not been shown in deployment. |
What would move it up
Release the code, trajectories and repaired graders, which would meet level 3. Level 4 needs a named team using the variance-aware protocol to select or compare agent changes in a real system. Replication of the candidate-level result on other benchmarks and models would also strengthen the claims.
The study
Yuanhao Zeng, Shaocheng Lan, Ao Lu et al.. The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements. arXiv, 17 November 2026. Read the paper
Notes
- Zeng et al., The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements. arXiv preprint, November 2026, CC BY 4.0. Source
- Project page named in the paper. The authors say reproducible materials will be released upon acceptance. Source
- Single-run spread: median standard deviation of 1.7 to 4.1 percentage points per benchmark. Over 60% of same-benchmark configuration pairs (217 of 341) differ by less than one single-run standard deviation.
- Literature survey: 171 results across nine papers, median margin 3.4 points (interquartile range 1.2 to 6.6). Runs needed at 80% power: about 5 (SpreadsheetBench), 10 (OfficeQA), 19 (ALFWorld), 23 (Terminal-Bench).
- Gain audit: one rule recovers a median 90% of the gain across 27 model-benchmark pairs above 3 points. Median gain falls from 17.6 to 0.8 points after repair.
- Selection pool: execution-based selection pairwise accuracy 49.7% (chance 50%). Regressions 30.5% of pool, 62.1% ± 42.9 of accepted candidates. LLM judges 45.4% to 49.0%.
- Re-evaluation: in-loop gains of 6.9 to 9.0 points against re-evaluated gains of +0.4 to -3.2 points. Averaging 16 validation rollouts: mean final test score 61.4% to 64.4%, best artifact 64.8% to 72.7% (Appendix F.3).
- SkillOpt-Lite code, one of the methods studied, as cited in the paper. Source
Edited and signed by
Stefan PerdalEditor and publisher85 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections


