# AI agents act on the wrong record when tools only count the others

A preprint tests dozens of AI agents on cancellations and deletions. Whether a search tool lists a second matching record, or only counts it, decides whether most of them stop.

- Category: Agents
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-16
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/ai-agents-act-on-the-wrong-record-when-tools-only-count-the-others

## The study

What LLM Agents See and What Users Say (preprint, not peer reviewed). arXiv 2611.17639. https://arxiv.org/abs/2611.17639

## At a glance

- What it shows: AI assistants that act for users often search a database first. When the search shows one match but says two exist, most of the tested assistants still act on the one they see. Listing both records stops most of them.
- What it does not show: It does not show how often this happens in real businesses. The tasks, the duplicate customer and the simulated users were all built by the researchers.
- Why it matters: Part of an assistant's safety sits in the tools it uses, not only in the AI model. How a search reports extra matches can decide whether the wrong customer's order is cancelled.

## What the study shows

The authors built two sets of tasks with 16 templates each. In the first, two records fit one description, such as two flight bookings for the same passenger name, and the agent is asked to cancel 'the' booking. In the second, the user makes a claim the record contradicts, such as 'Delete the folder. It is empty', while the folder holds one file. An agent here is an AI model that calls tools, such as search and delete, to act for a user.

They changed only what the search tool returned. It could list every match, show one match plus a count of the total, or show the first match alone. They ran 36 models from many vendors, called 'earlier agents', and three of the newest Claude and GPT models on selected cases. The measure was how often an agent carried out the permanent action instead of asking first.

The authors report that a count was not enough. Among 18 agents that act reliably when one record fits, about two thirds of runs went ahead when a second booking was only counted. About a quarter went ahead when both were listed. Runs where the agent opened the second record rarely acted, but the authors note this split is an association, since the agent chose whether to look.

Claims about the record in hand worked differently. When the user said a folder was empty and the record counted one item, a count stopped agents about as often as a full listing. Stating the claim also helped: across 27 agents, acting on a false claim fell from about two thirds of runs to under a third.

The newest Claude and GPT models opened every counted second record and then never acted. But when the search returned only the first match with no sign of others, they acted almost every time, just as with one record.

The authors then changed tau-bench, a public test of a retail customer-service agent. They added a second customer with the same name and postcode to 87 tasks. When the search returned that customer first, 13 agents wrote to the wrong account in about a quarter of conversations. They leaked that customer's details in nearly all of them.

Listing every match cut silent wrong-account writes by eight to ten times, but agents then read out the other customer's data in about a third of conversations. A lookup that asked for the user's street and only returned an account on a match kept task completion at the one-customer level, about two thirds. It produced no wrong writes or leaks, with a simulated user who gave the true street.

A survey of 418 lookup tools found that about two in five capped tools in agent toolkits and Model Context Protocol (MCP) servers, a common standard for plugging tools into agents, give no sign that more records exist. The paper is a preprint and has not been peer reviewed.

## Our reading

In our reading, the most useful finding is about tool design, not model choice. The newest models behaved well when a tool said more records existed. They behaved like everyone else when it said nothing. A model cannot be cautious about a record it has no reason to suspect.

Consider a support agent asked to refund 'the order for John Smith in postcode SW1'. If the customer search quietly returns the first John Smith, the refund may go to a stranger's account. The authors' results suggest a better model will not fix that. The search has to show the second John Smith, or demand a detail that tells them apart.

The authors also tested a line in the system prompt telling the agent to act only within what the user's words authorise. It cut acting a great deal, but behind a first-match search it cut acting about as much with one record as with two. In our reading, that makes it a general brake, not a check that finds the duplicate.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/ai-agents-act-on-the-wrong-record-when-tools-only-count-the-others

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
