# AI agents that remember past tool use fail when the tools quietly change

A preprint finds that agents reusing their own earlier tool calls do worse than newcomers after a silent interface change. They then report the broken task as done.

- Category: Agents
- Readiness level: I (Lab only) on Science Underway's four-step scale, as proposed
- Published: 2026-11-17
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/ai-agents-that-remember-past-tool-use-fail-when-the-tools-quietly-change

## The study

Experience Rot: Experienced Agents Fail Where Novices Succeed (preprint, not peer reviewed). arXiv 2611.20922. https://arxiv.org/abs/2611.20922

## At a glance

- What it shows: AI agents that remember how they used a tool before keep doing it the old way after the tool changes without warning. They succeed far less often than agents with no memory, and almost always say the job went fine.
- What it does not show: It does not show this happening in real products or real services. The authors built small test services and do not explain why the agents stop checking.
- Why it matters: Many agent products now keep memories so they can reuse what worked. This study suggests that memory can quietly turn into a source of errors that nobody sees.

## What the study shows

The authors built three small test services: a calendar, a task list and a mail client. Each has two versions that differ in one detail. For example, a field for meeting participants is renamed from "attendees" to "guests". The change is silent: the service accepts the old request, ignores what it does not recognise and replies that all went well.

Agents first did a task on one version, then a similar final task. The final task ran on the same version (valid experience), on the other version (out-of-date experience), or with no earlier task at all. Six language models were tested in a simple agent loop, across about 12,600 episodes. The current documentation was always one search away.

When the documentation gave a number's range but not its meaning, novice agents succeeded on 63% of final tasks. Agents with valid experience succeeded every time. Agents with out-of-date experience succeeded on only 23%. So stale experience was worse than none.

With full documentation, novices succeeded on 88% to 100% of tasks. Agents with out-of-date experience managed 26% to 49%. They failed even when the tool description in front of them listed the new field name. In 347 of 348 such failures, the agent told the user the task was done.

The authors report that experienced agents stopped looking things up: at most 6% searched the documentation before acting. More experience made it worse. After one, three and ten earlier tasks, success with stale experience fell from 46% to 33% to 26%. The effect appeared whether the experience sat in the conversation, in a saved memory file, in retrieved notes or in a reusable script.

Simple fixes mostly fell short. Announcing the update raised success only to 50%. Telling agents to read the documentation worked only when the instruction sat in the task message itself and the documentation held the answer. Asking agents to read back the stored result and fix mismatches worked best, at 91%, but cost more than twice the model calls.

## Our reading

In our reading, the most worrying part is not the failure rate but the silence. The service says yes, the agent says done, and the stored record is wrong. A practical example: a scheduling agent books a review meeting, the calendar provider renames a field, and the meeting is saved with no participants. The user is told it is booked.

The design is careful. Valid and out-of-date arms use the same tasks and differ only in which version the agent learned on. Most complete-documentation conditions were registered in advance, and the authors report that every advance prediction held. That makes the core result harder to dismiss as a fluke of analysis.

This is a preprint and has not been peer reviewed. On our four-level readiness scale it sits at level I, lab only. The tests run on the authors' own synthetic services, not a public benchmark, and we found no released code or data.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/ai-agents-that-remember-past-tool-use-fail-when-the-tools-quietly-change

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
