# Coding agents finish most tasks alone but only half of task chains

A new benchmark from Amazon researchers chains 20 coding requests together. Ten agents complete about three quarters of the single requests but only about half of the chains, mostly because they build on mistakes they never notice.

- Category: Code
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-16
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/coding-agents-finish-most-tasks-alone-but-only-half-of-task-chains

## The study

Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.02207, arXiv 2611.02207. https://arxiv.org/abs/2611.02207

## At a glance

- What it shows: When AI coding assistants handle a series of linked jobs without anyone checking their work, an unnoticed mistake early on makes later jobs fail much more often. A short outside message saying 'this failed' recovers part of the loss.
- What it does not show: It does not show how these agents behave in real companies, on languages other than Python, or with a human reviewer. The 'failed' message came from hidden tests, not from a real user.
- Why it matters: Teams increasingly hand AI agents a sequence of tasks and stop reviewing each one. This study suggests the checks around the agent matter as much as the agent itself.

## What the study shows

The authors built a benchmark called BOBA-Bench. It turns the commit histories of 140 open-source Python projects into 280 chains of 20 consecutive coding requests. Each request builds on the code left by the one before. Hidden tests, written by the original developers, grade every step. The agent never sees the verdict.

Ten coding agents were tested, including models from Anthropic, OpenAI, Zhipu AI and Moonshot AI. Each request ran in a fresh conversation, so the agent remembered nothing of earlier steps. The authors report that the agents completed 66% to 84% of individual requests. Yet they completed only about half of the full chains on average. The best model finished about two thirds of chains, the weakest under two fifths.

The authors then asked why. Code written by an agent was not the main problem. When earlier steps had been verified as correct, the next request succeeded almost as often on agent-written code as on the developers' own code, about 1 point lower on average.

The problem was unverified mistakes. Agents declared a request done once their own tests passed, even when hidden tests failed. The authors call these silent misses. After one, the next request failed noticeably more often, and the odds of failure roughly doubled with each extra open mistake. Only 14% to 33% of tests that broke were ever fixed later.

Agents also proved poor reviewers of each other's work. Some accepted most wrong code. Others rejected about half of correct code. None did much better than chance at telling the two apart.

Finally, the authors sent a one-line 'verification failed' message after each failed request, with one chance to repair. Average chain completion rose from about 51% to 59%. Gains ranged from 1 to 22 points depending on the model. The repair round also broke previously working code more often, and added extra model calls.

## Our reading

In our reading, the most useful result is the split between authorship and verification. A common worry is that agent-written code is messy and hard to build on. Here, once checked, it held up nearly as well as human code. The damage came from mistakes nobody caught. That moves the question from 'is the agent good enough?' to 'who checks its work, and when?'.

The self-testing result deserves attention. Agents ran tests before almost every submission. A green local run barely changed the miss rate. In practice, an agent saying 'all tests pass' tells a team very little. A concrete example: an agent asked to add a feature to a library rewrote a function it had added one step earlier, because it no longer remembered why. Its own tests passed. Two earlier tests broke and stayed broken for the rest of the chain.

This is a preprint and has not been peer reviewed. On our four-level readiness scale we rate it III, open source available. It reports results on a new public benchmark, and the authors say the benchmark, grader and run records are released on GitHub. We found no documented use by a named organisation, so it does not reach level IV.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/coding-agents-finish-most-tasks-alone-but-only-half-of-task-chains

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
