# Thinking harder makes AI shopping agents more confident, not more accurate

A preprint tests ten commercial AI models on product choice. Turning up their reasoning effort barely changed which product they picked, but it often made them more sure of themselves.

- Category: Safety & evaluation
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-18
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/thinking-harder-makes-ai-shopping-agents-more-confident-not-more-accurate

## The study

Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents (preprint, not peer reviewed). arXiv 2611.22033. https://arxiv.org/abs/2611.22033

## At a glance

- What it shows: Asking AI shopping assistants to think harder did not reliably help them pick the right product. It did make most of them more confident. Most models were already too confident, and they tended to make the same mistakes as each other.
- What it does not show: It does not test real shoppers, real purchases, or agents that search and browse a store. It also does not explain why the extra confidence appears or why models share their mistakes.
- Why it matters: Shopping agents are meant to ask the user when they are unsure. If their confidence rises without their accuracy, that safety check fails, and asking a second model for its view helps less than expected.

## What the study shows

The authors built three shopping tests from public product data: WebShop and SQID, both from Amazon listings, and WANDS, from the furniture retailer Wayfair. Each task gave a model a customer request and ten candidate products. Exactly one product was correct. The other nine were near misses, such as the same item in the wrong size or colour. Search and browsing were removed, so only the final choice was judged.

Ten commercial models from six providers were run at low, medium and high reasoning effort. Reasoning effort is a setting that asks a model to spend more internal 'thinking' before it answers. Each model also rated its confidence from 0 to 10. The authors compared each model with itself, task by task, across effort levels.

The authors report that moving from low to high effort rarely changed accuracy in a reliable way. After correcting for the many comparisons made, only one model on one test improved: Qwen 3.8 Max on WebShop, by about 5 points. In the same move, stated confidence rose significantly for most models on WebShop and for about half on the other two tests. The rises were small, but they mostly started from levels that were already too high.

Calibration means how well stated confidence matches how often a model is actually right. It depended on the model, not on the effort setting. The two Claude models were close to calibrated on two of the tests. The other eight were overconfident by roughly 10 to 30 points, depending on the test.

Models also failed together. On WebShop at low effort, all ten models got the same 47 tasks wrong. If their errors had been independent, almost no task would be failed by all ten. When at least two models were wrong, over 80% of their wrong picks landed on the same product.

The paper is a preprint and has not been peer reviewed.

## Our reading

In our reading, the headline result is less about accuracy than about honesty. A shopping agent's confidence is what lets it hand a decision back to a person. This study suggests that the effort setting can push that signal in the wrong direction. For example, an agent asked for a 2.6 ounce deodorant might pick the near-identical 3.4 ounce one. If it rates that choice 10 out of 10, a rule like 'ask the user when confidence is below 8' never fires.

The shared-error finding matters just as much. A common safeguard is to ask a second model and act only when both agree. Here, agreement between frontier models was weak evidence of a correct choice, because they were drawn to the same plausible wrong listing. Reasoning effort did not reduce this.

We rate the work at level II, Benchmarked, on our four-level scale. It reports results on tasks built from public benchmarks with careful paired statistics. Code, task files and logs are promised for release on publication but are not yet available, so level III is not met.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/thinking-harder-makes-ai-shopping-agents-more-confident-not-more-accurate

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
