AI agents act on the wrong record when tools only count the others

A preprint tests dozens of AI agents on cancellations and deletions. Whether a search tool lists a second matching record, or only counts it, decides whether most of them stop.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
A studio still life with a wooden index box containing cream cards, with a tiny blue tab peeking from behind.AI illustration
Illustration: AI generated

What the study shows

The authors built two sets of tasks with 16 templates each. In the first, two records fit one description, such as two flight bookings for the same passenger name, and the agent is asked to cancel 'the' booking. In the second, the user makes a claim the record contradicts, such as 'Delete the folder. It is empty', while the folder holds one file. An agent here is an AI model that calls tools, such as search and delete, to act for a user.

They changed only what the search tool returned. It could list every match, show one match plus a count of the total, or show the first match alone. They ran 36 models from many vendors, called 'earlier agents', and three of the newest Claude and GPT models on selected cases. The measure was how often an agent carried out the permanent action instead of asking first.

The authors report that a count was not enough. Among 18 agents that act reliably when one record fits, about two thirds of runs went ahead when a second booking was only counted. About a quarter went ahead when both were listed. Runs where the agent opened the second record rarely acted, but the authors note this split is an association, since the agent chose whether to look.

Claims about the record in hand worked differently. When the user said a folder was empty and the record counted one item, a count stopped agents about as often as a full listing. Stating the claim also helped: across 27 agents, acting on a false claim fell from about two thirds of runs to under a third.

The newest Claude and GPT models opened every counted second record and then never acted. But when the search returned only the first match with no sign of others, they acted almost every time, just as with one record.

The authors then changed tau-bench, a public test of a retail customer-service agent. They added a second customer with the same name and postcode to 87 tasks. When the search returned that customer first, 13 agents wrote to the wrong account in about a quarter of conversations. They leaked that customer's details in nearly all of them.

Listing every match cut silent wrong-account writes by eight to ten times, but agents then read out the other customer's data in about a third of conversations. A lookup that asked for the user's street and only returned an account on a match kept task completion at the one-customer level, about two thirds. It produced no wrong writes or leaks, with a simulated user who gave the true street.

A survey of 418 lookup tools found that about two in five capped tools in agent toolkits and Model Context Protocol (MCP) servers, a common standard for plugging tools into agents, give no sign that more records exist. The paper is a preprint and has not been peer reviewed.

Our reading

In our reading, the most useful finding is about tool design, not model choice. The newest models behaved well when a tool said more records existed. They behaved like everyone else when it said nothing. A model cannot be cautious about a record it has no reason to suspect.

Consider a support agent asked to refund 'the order for John Smith in postcode SW1'. If the customer search quietly returns the first John Smith, the refund may go to a stranger's account. The authors' results suggest a better model will not fix that. The search has to show the second John Smith, or demand a detail that tells them apart.

The authors also tested a line in the system prompt telling the agent to act only within what the user's words authorise. It cut acting a great deal, but behind a first-match search it cut acting about as much with one record as with two. In our reading, that makes it a general brake, not a check that finds the duplicate.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetThe work is a lab study with built tasks, a theory framing and a modified benchmark.
II  BenchmarkedMetResults are reported on 36 earlier agents across 16-template grids, and on tau-bench retail with a second customer added. These are the authors' own benchmark variants, not a shared public leaderboard, but they are measured and compared across many models.
III  Open sourceMetThe paper states that code, prompts, fixtures, rollouts and the scripts that regenerate every table are released. No URL appears in the text provided, so this rests on the authors' statement.
IV  In productionNot yetNo documented use by a named organisation in production. The street check was tested only in a simulated retail benchmark with a model-played user.

What would move it up
To reach level 4, a named organisation would need to deploy list-or-check lookups, or a standard 'more results exist' flag, in a live agent and report outcomes. Useful evidence would include tests with users who lie about or do not know the check detail, with real customers, and on other domains. Replication by independent groups on the released code would also strengthen level 2.

The study

Yue Zhao, Angela Huyue Zhang, S. Alex Yang et al.. What LLM Agents See and What Users Say. arXiv, 13 November 2026. Read the paper

Preprint, not peer reviewed · Licence arXiv (no reuse)

Notes

  1. Zhao, Zhang, Yang, Wei and Hu, 'What LLM Agents See and What Users Say', arXiv preprint, November 2026. Not peer reviewed. Source
  2. The authors state that code, prompts, fixtures, rollouts and table scripts are released. No link appears in the paper text.
  3. Which-record tasks, 18 agents that act reliably on one record: 65% acting with a count against 24% with both listed (Table 2). Across all 36 earlier agents, 23 acted in over half of runs under a count, 6 under a listing (Table 1).
  4. Empty-claim tasks, 27 agents: 28% acting with a count against 29% with a listing when the claim is stated. Acting on a false claim fell from 67% to 28% when the claim was stated (Table 2, Figure 4).
  5. tau-bench with the second customer first: wrong-account writes in 23.2% (original user) and 25.8% (street-knowing user) of conversations, full-value leaks in about 92%. Street check completion 66.5% against 66.0% with one customer (Table 3).
  6. Survey: 85 of 211 capped MCP and toolkit tools (40%) and all 82 capped benchmark tools give no sign of more records (Figure A8).

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Agents