# AI peer reviewers tested on planted errors: model choice matters most

A new benchmark plants known mistakes in real scientific papers and counts how many AI reviewers catch. The model used makes the biggest difference; extra tools help only where the model has a gap.

- Category: Safety & evaluation
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-13
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/ai-peer-reviewers-tested-on-planted-errors-model-choice-matters-most

## The study

Auditing AI peer reviewers: a benchmark on real scientific papers (preprint, not peer reviewed). arXiv 2611.17052. https://arxiv.org/abs/2611.17052

## At a glance

- What it shows: The researchers planted known mistakes in 30 real research papers and checked which AI reviewing tools found them. The choice of AI model changed the catch rate far more than anything else. Making the tool fetch and read the papers it cites helped a lot in astrophysics but not in machine learning.
- What it does not show: It does not show whether these tools judge a paper's originality or importance, or how they perform on real journal submissions. The test used planted mistakes, not natural ones, in a small set of papers.
- Why it matters: AI is already used to review research at scale, but there have been few fair ways to compare tools. This gives buyers and journals a repeatable test that also counts false alarms.

## What the study shows

The authors built a test for AI peer reviewers, meaning AI systems that read a scientific paper and write a critique. They took 15 astrophysics and 15 machine learning (ML) preprints from arXiv, a public archive of research papers. An AI agent then planted errors into the papers' source files: 357 in total.

The errors came in four types: general issues such as contradictions, mathematics, numbers, and misquoted citations. Each type came at three levels, from subtle to blatant. Each paper was also reviewed in a clean copy, to count false alarms. A false alarm, or false positive, is a flagged problem that is not real.

Twelve reviewers were tested. They included three leading AI models given a one-line prompt or a long, structured prompt. They also included Claude Code, Anthropic's coding agent, which can run code and fetch papers. Three public reviewers were included, plus the authors' own system, SKEPTHICAL. SKEPTHICAL splits the job across specialist agents. It checks maths with computer algebra (software that does symbolic maths) and numbers with code. It also downloads and reads each cited paper.

The authors report that model choice mattered most. With the same one-line prompt, the three models differed by about 50 percentage points in errors caught, on both subjects. The long prompt narrowed that gap, mainly by lifting the weaker models.

On astrophysics, SKEPTHICAL caught about 71% of errors, against about 61% for the best single model. The authors call this gap significant. On ML the best single model scored slightly higher, and the difference was not significant. SKEPTHICAL's clearest lead was on citations in astrophysics, where it caught nearly half and no other system passed about a quarter.

Most systems raised false alarms on under 10% of their flags. Maths flags were the least reliable and made up half of all false alarms. Two small open models trained on review data caught under 3% of errors. The work is a preprint and has not been peer reviewed.

## Our reading

In our reading, the most useful result is not SKEPTHICAL's score but the size of the gap between models. The same instruction produced a catch rate of about 61% with one model and about 12% with another. For anyone choosing a review tool, the underlying model is the first question to ask.

The second lesson is about tools. Giving an agent the ability to fetch cited papers did nothing on its own. Claude Code had that access but, according to the execution logs, never used it. SKEPTHICAL forced the step, and that is where its citation lead came from. In ML, where the models seem to know the literature already, forcing retrieval added little. A tool helps when it supplies something the model lacks.

We rate this work at level III, open source available. The evaluation pipeline, planted errors and judging records are released publicly. SKEPTHICAL itself can be used online, but the paper does not say its code is released. We found no named organisation using it in production.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/ai-peer-reviewers-tested-on-planted-errors-model-choice-matters-most

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
