Coding agents finish most tasks alone but only half of task chains
A new benchmark from Amazon researchers chains 20 coding requests together. Ten agents complete about three quarters of the single requests but only about half of the chains, mostly because they build on mistakes they never notice.


What the study shows
The authors built a benchmark called BOBA-Bench. It turns the commit histories of 140 open-source Python projects into 280 chains of 20 consecutive coding requests. Each request builds on the code left by the one before. Hidden tests, written by the original developers, grade every step. The agent never sees the verdict.
Ten coding agents were tested, including models from Anthropic, OpenAI, Zhipu AI and Moonshot AI. Each request ran in a fresh conversation, so the agent remembered nothing of earlier steps. The authors report that the agents completed 66% to 84% of individual requests. Yet they completed only about half of the full chains on average. The best model finished about two thirds of chains, the weakest under two fifths.
The authors then asked why. Code written by an agent was not the main problem. When earlier steps had been verified as correct, the next request succeeded almost as often on agent-written code as on the developers' own code, about 1 point lower on average.
The problem was unverified mistakes. Agents declared a request done once their own tests passed, even when hidden tests failed. The authors call these silent misses. After one, the next request failed noticeably more often, and the odds of failure roughly doubled with each extra open mistake. Only 14% to 33% of tests that broke were ever fixed later.
Agents also proved poor reviewers of each other's work. Some accepted most wrong code. Others rejected about half of correct code. None did much better than chance at telling the two apart.
Finally, the authors sent a one-line 'verification failed' message after each failed request, with one chance to repair. Average chain completion rose from about 51% to 59%. Gains ranged from 1 to 22 points depending on the model. The repair round also broke previously working code more often, and added extra model calls.
Our reading
In our reading, the most useful result is the split between authorship and verification. A common worry is that agent-written code is messy and hard to build on. Here, once checked, it held up nearly as well as human code. The damage came from mistakes nobody caught. That moves the question from 'is the agent good enough?' to 'who checks its work, and when?'.
The self-testing result deserves attention. Agents ran tests before almost every submission. A green local run barely changed the miss rate. In practice, an agent saying 'all tests pass' tells a team very little. A concrete example: an agent asked to add a feature to a library rewrote a function it had added one step earlier, because it no longer remembered why. Its own tests passed. Two earlier tests broke and stayed broken for the rest of the chain.
This is a preprint and has not been peer reviewed. On our four-level readiness scale we rate it III, open source available. It reports results on a new public benchmark, and the authors say the benchmark, grader and run records are released on GitHub. We found no documented use by a named organisation, so it does not reach level IV.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The work is a benchmark and an analysis, and its claims are backed by controlled experiments on 280 chains. |
| II Benchmarked | Met | It reports results for ten named models on a new public benchmark, with repository-clustered confidence intervals and pairwise tests (Tables 2, 25, 32). It is not compared to an established external leaderboard. |
| III Open source | Met | The authors say the benchmark, construction pipeline, specifications, hidden contracts, gold patches, sealed images, trajectories, grader and analysis scripts are at https://github.com/amazon-agi/BOBA-bench. Source code stays under each repository's licence. We could not inspect the repo from the paper alone. |
| IV In production | Not yet | No documented production use by a named organisation. The authors say the benchmark measures fixed-test correctness, not production readiness. The verification recommendations are described as patches, not a validated product. |
What would move it up
To reach level 4, a named organisation would need to adopt the benchmark or the verification-gate approach in a real workflow and document the results. Independent replication by another group, a human review of the specification quality, and tests on non-Python and larger codebases would also strengthen it. The next step for the research itself is an agent that can tell its wrong work from its right work without an oracle.
The study
Alejandro Cuadron, Arpit Gupta. Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone. arXiv, 1 November 2026. doi:10.48550/arxiv.2611.02207
Notes
- Cuadron, A. and Gupta, A. Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone. arXiv preprint, 1 November 2026. CC BY 4.0. Source
- Benchmark, grader, specifications and run records, as stated by the authors. Not inspected by us. Source
- Exact figures: average pass@1 51% (eight attempts per chain); range 39.2% (Claude Opus 4.8) to 66.7% (Claude Opus 5.5); strict pass@1 average 39%; request completion 66.2% to 84.0% (Table 2).
- Authorship gap on verified code: minus 3.6 to plus 0.8 points across eleven models, average drop about 1 point (Section 4.2, Tables 9 and 10).
- Verification effect: next-request failure up about 18 points after one silent miss; 9 to 19 points against the developers' state on the same requests (Fable 5 interval spans zero); odds ratio per extra open defect 1.72 to 2.32 (Tables 21 and 22).
- Review experiment: lenient reviewers accepted 77% to 95% of wrong code; strict reviewers rejected 41% to 58% of correct code; Youden J at most 0.15 (Table 12).
- Verified arm: pass@1 gains 1.1 to 22.1 points, seven of ten intervals exclude zero; regression rate up 3.2 to 8.9 points; 47 to 294 extra calls per chain (Table 41).
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

