# Research agents stumble when asked to continue real, messy projects

A new benchmark built from real researcher sessions finds the best of seven models fully completes only two-thirds of tasks. Scores fall sharply once earlier decisions and existing files are checked.

- Category: Agents
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-12
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/research-agents-stumble-when-asked-to-continue-real-messy-projects

## The study

ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.14364, arXiv 2611.14364. https://arxiv.org/abs/2611.14364

## At a glance

- What it shows: On 75 tasks taken from real, unfinished research projects, the strongest of seven models met every checklist item in 66.2% of runs and the weakest in 25.8%. Three mid-ranked models scored about 69% on basic delivery checks but 28.4% to 40.0% once requirements from earlier in the project were checked.
- What it does not show: It does not show how these scores relate to agents' day-to-day usefulness for researchers, and the scoring is done by another AI model rather than human reviewers. Each model was also tested inside a particular agent tool, so model and tool effects are mixed.
- Why it matters: Benchmarks that start from a clean slate may overstate how reliably agents can pick up a step in an ongoing project. Checking whether an agent respects earlier decisions changes which models look good.

## What the study shows

The paper, a preprint that has not been peer reviewed, comes from the Shanghai Artificial Intelligence Laboratory, Fudan University and four other Chinese universities. It introduces ARE-bench, a test of whether AI research agents can carry out one requested step inside a project that is already under way. Most existing research-agent benchmarks give an agent a self-contained goal, such as replicating a paper, in a tidy starting folder.

The authors collected agent sessions and full project folders from recruited researchers. They cut each session at a real user request, such as fixing a failed training run or restructuring a paper. The agent receives a summary of earlier requests still in force (10.3 on average), the current request and the project folder replayed exactly as it was at that point, without cleanup. Folders hold 41.1 files on average, and 65.3% of tasks contain files written by agents in earlier turns, including outdated drafts and paused experiments.

The 75 tasks span seven research stages: survey, design, data and environment setup, code, execution, analysis and writing. They compile into 1,173 yes/no checklist items, about 15.6 per task. Items are meant to credit any valid solution rather than one fixed answer.

Seven models were each run three times per task. The main score, instance success rate (ISR), counts a task as done only if every checklist item passes. The authors report ISR of 66.2% for GPT-6.1-Sol, 56.9% for Claude Opus 5.5, 39.6% for DeepSeek-V4.1-Flash, 37.8% for Qwen3.8-Max, 30.7% for GLM-5.3-Flash, 28.0% for Kimi-K3 and 25.8% for Qwen3.8-27B.

A partial-credit score, the share of checklist items passed, ranged only from 79.7% to 92.7%. So models do most of the work but often miss something. Of GPT-6.1-Sol's 76 failed runs, 47 missed only one or two items.

Correctness was the weakest area for every model. Passing all correctness items happened in 31.1% to 71.1% of runs, compared with 72.0% to 87.6% for completeness. Between 85.5% and 92.8% of failed runs had at least one correctness error.

The authors then re-scored the same outputs with a narrower checklist. Checking only the explicit delivery terms and general professional norms, DeepSeek-V4.1-Flash, GLM-5.3-Flash and Kimi-K3 each scored about 69%. Adding requirements inherited from earlier session decisions and from project materials cut them to 40.0%, 31.6% and 28.4%. The gap between DeepSeek and Kimi grew from 0.44 points to 11.56.

Design and Execution were among the three hardest stages for every model, with average strict success of 59.5% and 57.4%. Data and environment setup was easiest at 86.4%. Rankings shifted by stage: Kimi-K3 ranked third on code but last on writing.

Cost did not follow success. At the authors' fixed price estimates, GPT-6.1-Sol cost about USD 2.17 per run and Claude Opus 5.5 about USD 5.69, roughly 2.6 times as much, for an ISR 9.3 points lower.

## Our reading

In our reading, the most useful result is the re-scoring experiment. Holding the agents' outputs fixed and only changing what is checked removes a lot of noise. Three models that look identical on surface checks separate clearly once the test asks whether they kept to earlier decisions and the method already in the project. That is a direct argument that narrow checks can hide real differences.

The single worked example makes the point concrete. In a task to repair a stalled training run, two agent runs both got training going again with non-zero loss. One passed all 20 checks. The other passed 16 and had quietly changed coordinate handling, labelling and loss functions away from the paper's method. A test that only asked whether training resumed would have scored both as successes.

On readiness, we rate this level II, Benchmarked. The authors report results for seven models on their own benchmark. A project page is listed, but the paper does not say the tasks, workspaces or scoring code are released, or under what licence, so level III is not yet met.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/research-agents-stumble-when-asked-to-continue-real-messy-projects

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
