Thinking harder makes AI shopping agents more confident, not more accurate
A preprint tests ten commercial AI models on product choice. Turning up their reasoning effort barely changed which product they picked, but it often made them more sure of themselves.

AI illustrationWhat the study shows
The authors built three shopping tests from public product data: WebShop and SQID, both from Amazon listings, and WANDS, from the furniture retailer Wayfair. Each task gave a model a customer request and ten candidate products. Exactly one product was correct. The other nine were near misses, such as the same item in the wrong size or colour. Search and browsing were removed, so only the final choice was judged.
Ten commercial models from six providers were run at low, medium and high reasoning effort. Reasoning effort is a setting that asks a model to spend more internal 'thinking' before it answers. Each model also rated its confidence from 0 to 10. The authors compared each model with itself, task by task, across effort levels.
The authors report that moving from low to high effort rarely changed accuracy in a reliable way. After correcting for the many comparisons made, only one model on one test improved: Qwen 3.8 Max on WebShop, by about 5 points. In the same move, stated confidence rose significantly for most models on WebShop and for about half on the other two tests. The rises were small, but they mostly started from levels that were already too high.
Calibration means how well stated confidence matches how often a model is actually right. It depended on the model, not on the effort setting. The two Claude models were close to calibrated on two of the tests. The other eight were overconfident by roughly 10 to 30 points, depending on the test.
Models also failed together. On WebShop at low effort, all ten models got the same 47 tasks wrong. If their errors had been independent, almost no task would be failed by all ten. When at least two models were wrong, over 80% of their wrong picks landed on the same product.
The paper is a preprint and has not been peer reviewed.
Our reading
In our reading, the headline result is less about accuracy than about honesty. A shopping agent's confidence is what lets it hand a decision back to a person. This study suggests that the effort setting can push that signal in the wrong direction. For example, an agent asked for a 2.6 ounce deodorant might pick the near-identical 3.4 ounce one. If it rates that choice 10 out of 10, a rule like 'ask the user when confidence is below 8' never fires.
The shared-error finding matters just as much. A common safeguard is to ask a second model and act only when both agree. Here, agreement between frontier models was weak evidence of a correct choice, because they were drawn to the same plausible wrong listing. Reasoning effort did not reduce this.
We rate the work at level II, Benchmarked, on our four-level scale. It reports results on tasks built from public benchmarks with careful paired statistics. Code, task files and logs are promised for release on publication but are not yet available, so level III is not met.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The work is a controlled lab-style evaluation on constructed tasks with no deployment. |
| II Benchmarked | Met | Results come from three public benchmarks (WebShop, SQID, WANDS) with ten models, paired statistics and Holm correction. The task sets are the authors' own constructions, not standard leaderboard splits. |
| III Open source | Not yet | The reproducibility section says task files, logs and code will be released later upon publication. No link is given. |
| IV In production | Not yet | No named organisation uses the findings or the evaluation in production. |
What would move it up
Releasing the task files, response logs and analysis code would reach level 3. Level 4 would need documented use by a named organisation, for example a shopping-agent team adopting the evaluation. Repeat runs, and tests with search and navigation included, would strengthen the findings.
The study
Adi Krishnamoorthy, X. Y. Han. Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents. arXiv, 17 November 2026. Read the paper
Notes
- Krishnamoorthy, A. and Han, X. Y. Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents. arXiv preprint, November 2026, not peer reviewed. Licence CC BY NC SA 4.0. Source
- Code, task files and response logs: the authors say these will be released on publication. No link was available at the time of writing.
- Test sizes: 496 WebShop tasks, 500 SQID tasks and 330 WANDS tasks, about 37,100 valid responses in total. Success ranges across all conditions: 74.0 to 88.1% (WebShop), 43.7 to 50.6% (SQID), 65.8 to 73.7% (WANDS).
- Only accuracy gain surviving Holm correction for low to high effort: Qwen 3.8 Max on WebShop, +5.5 points. GPT-6 Luna on WANDS survived correction for low to medium, +5.5 points. 8 of 11 nominally significant accuracy effects disappeared under correction (Supplement Table S2).
- Overconfidence, pooled over effort: Claude Fable 5.1 and Claude Opus 5.5 were -2.8 and -6.7 points on WebShop, +12.4 and +10.8 on SQID, +0.7 and -1.5 on WANDS. The other eight models were +7 to +16, +27 to +33 and +10 to +20 points on the same tests.
- Shared errors on WebShop at low effort: 319 of 493 tasks solved by all ten models (about 52 expected under independence) and 47 failed by all ten (about 0 expected). Implied pairwise error correlation 0.70 to 0.81 across tests. Share of wrong picks on the most popular wrong product: 81 to 87%, against 32 to 36% if random.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

