AI agents that remember past tool use fail when the tools quietly change
A preprint finds that agents reusing their own earlier tool calls do worse than newcomers after a silent interface change. They then report the broken task as done.

AI illustrationWhat the study shows
The authors built three small test services: a calendar, a task list and a mail client. Each has two versions that differ in one detail. For example, a field for meeting participants is renamed from "attendees" to "guests". The change is silent: the service accepts the old request, ignores what it does not recognise and replies that all went well.
Agents first did a task on one version, then a similar final task. The final task ran on the same version (valid experience), on the other version (out-of-date experience), or with no earlier task at all. Six language models were tested in a simple agent loop, across about 12,600 episodes. The current documentation was always one search away.
When the documentation gave a number's range but not its meaning, novice agents succeeded on 63% of final tasks. Agents with valid experience succeeded every time. Agents with out-of-date experience succeeded on only 23%. So stale experience was worse than none.
With full documentation, novices succeeded on 88% to 100% of tasks. Agents with out-of-date experience managed 26% to 49%. They failed even when the tool description in front of them listed the new field name. In 347 of 348 such failures, the agent told the user the task was done.
The authors report that experienced agents stopped looking things up: at most 6% searched the documentation before acting. More experience made it worse. After one, three and ten earlier tasks, success with stale experience fell from 46% to 33% to 26%. The effect appeared whether the experience sat in the conversation, in a saved memory file, in retrieved notes or in a reusable script.
Simple fixes mostly fell short. Announcing the update raised success only to 50%. Telling agents to read the documentation worked only when the instruction sat in the task message itself and the documentation held the answer. Asking agents to read back the stored result and fix mismatches worked best, at 91%, but cost more than twice the model calls.
Our reading
In our reading, the most worrying part is not the failure rate but the silence. The service says yes, the agent says done, and the stored record is wrong. A practical example: a scheduling agent books a review meeting, the calendar provider renames a field, and the meeting is saved with no participants. The user is told it is booked.
The design is careful. Valid and out-of-date arms use the same tasks and differ only in which version the agent learned on. Most complete-documentation conditions were registered in advance, and the authors report that every advance prediction held. That makes the core result harder to dismiss as a fluke of analysis.
This is a preprint and has not been peer reviewed. On our four-level readiness scale it sits at level I, lab only. The tests run on the authors' own synthetic services, not a public benchmark, and we found no released code or data.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | A controlled lab study on synthetic services built by the authors, with six models and a simple agent loop. |
| II Benchmarked | Not yet | Results are on the authors' own bespoke test, not a public benchmark that others can run and compare against. There is no evidence that a shared benchmark or leaderboard exists. The preregistration is a point in favour of rigour but not a public benchmark. |
| III Open source | Not yet | The text we read gives no link to code, data or the preregistered analysis scripts, though it mentions analysis code and hashes. Any release would need to be checked. |
| IV In production | Not yet | No documented use by a named organisation. Two authors are at eBay, but the paper reports no production deployment or production data. |
What would move it up
To reach level 2, the authors could publish the test as a public benchmark and have other groups run it on other models. To reach level 3, they would release the services, tasks, prompts and analysis code. To reach level 4, a named organisation would need to document the failure, or a fix such as read-back checks, in a live agent system. Tests on real services, with real memory products and in agent frameworks with their own checks, would also strengthen the case.
The study
Zhen Xu, Hang Yin, Zitao Zhang et al.. Experience Rot: Experienced Agents Fail Where Novices Succeed. arXiv, 16 November 2026. Read the paper
Notes
- Xu, Yin, Zhang, Zhao and Liu, "Experience Rot: Experienced Agents Fail Where Novices Succeed", arXiv preprint 2611.20922, 16 November 2026, not peer reviewed. Licence CC BY 4.0. Source
- Headline figures: with incomplete documentation, success was 0.63 with no experience, 1.00 with valid experience and 0.23 with out-of-date experience (six models, four changes). With complete documentation, out-of-date success was 0.26 to 0.49.
- Remedies (complete documentation, out-of-date experience): nothing added 0.46; version and notice 0.50; read-docs instruction in system prompt 0.29; same in task message 0.99; verify and correct 0.91 at 6.5 model calls against 2.9 with valid experience; write returns stored object 0.79 against 0.45 baseline.
- We found no link to code, data or preregistration materials in the paper.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections
