AI peer reviewers tested on planted errors: model choice matters most

A new benchmark plants known mistakes in real scientific papers and counts how many AI reviewers catch. The model used makes the biggest difference; extra tools help only where the model has a gap.

5 min readIIIOpen sourcePreprint, not peer reviewed
Save
Figure 1 from the paper "Auditing AI peer reviewers: a benchmark on real scientific papers"
Detection rate for each reviewer rises or falls as planted errors go from subtle to blatant, shown separately for each error type and for the astrophysics and machine learning papers. Figure 1 from Íñigo Zubeldia et al., 2026, arXiv:2611.17052. CC BY 4.0.

What the study shows

The authors built a test for AI peer reviewers, meaning AI systems that read a scientific paper and write a critique. They took 15 astrophysics and 15 machine learning (ML) preprints from arXiv, a public archive of research papers. An AI agent then planted errors into the papers' source files: 357 in total.

The errors came in four types: general issues such as contradictions, mathematics, numbers, and misquoted citations. Each type came at three levels, from subtle to blatant. Each paper was also reviewed in a clean copy, to count false alarms. A false alarm, or false positive, is a flagged problem that is not real.

Twelve reviewers were tested. They included three leading AI models given a one-line prompt or a long, structured prompt. They also included Claude Code, Anthropic's coding agent, which can run code and fetch papers. Three public reviewers were included, plus the authors' own system, SKEPTHICAL. SKEPTHICAL splits the job across specialist agents. It checks maths with computer algebra (software that does symbolic maths) and numbers with code. It also downloads and reads each cited paper.

The authors report that model choice mattered most. With the same one-line prompt, the three models differed by about 50 percentage points in errors caught, on both subjects. The long prompt narrowed that gap, mainly by lifting the weaker models.

On astrophysics, SKEPTHICAL caught about 71% of errors, against about 61% for the best single model. The authors call this gap significant. On ML the best single model scored slightly higher, and the difference was not significant. SKEPTHICAL's clearest lead was on citations in astrophysics, where it caught nearly half and no other system passed about a quarter.

Most systems raised false alarms on under 10% of their flags. Maths flags were the least reliable and made up half of all false alarms. Two small open models trained on review data caught under 3% of errors. The work is a preprint and has not been peer reviewed.

Our reading

In our reading, the most useful result is not SKEPTHICAL's score but the size of the gap between models. The same instruction produced a catch rate of about 61% with one model and about 12% with another. For anyone choosing a review tool, the underlying model is the first question to ask.

The second lesson is about tools. Giving an agent the ability to fetch cited papers did nothing on its own. Claude Code had that access but, according to the execution logs, never used it. SKEPTHICAL forced the step, and that is where its citation lead came from. In ML, where the models seem to know the literature already, forcing retrieval added little. A tool helps when it supplies something the model lacks.

We rate this work at level III, open source available. The evaluation pipeline, planted errors and judging records are released publicly. SKEPTHICAL itself can be used online, but the paper does not say its code is released. We found no named organisation using it in production.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetThe work goes beyond lab concept. A working system and a full evaluation pipeline exist.
II  BenchmarkedMetTwelve systems were benchmarked on 30 real arXiv preprints with 357 injected errors. Results come with bootstrap confidence intervals and a one-paper human check. The benchmark is the authors' own, not an established public one.
III  Open sourceMetThe evaluation pipeline, paper lists, injection catalogues and adjudication logs are released at github.com/inigozubeldia/skepthical-benchmark. SKEPTHICAL is publicly usable at skepthical-ai.org. SKEPTHICAL's own source code and weights are not stated to be released, so this level rests on the benchmark release. I could not verify the repository or its licence.
IV  In productionNot yetThe paper names no organisation using SKEPTHICAL in production and reports no real-world outcomes. The claim that LLM review is at production scale is cited to other systems, not shown for this one.

What would move it up
To reach Level 4, a named journal, conference or lab would need to document using SKEPTHICAL or the benchmark on real submissions, with outcomes. Useful extra evidence: confirmation that SKEPTHICAL's code is open, larger and more varied corpora, a human-checked false-positive panel, and an equal-cost single-provider ensemble for comparison.

Checked automatically every week, last on 16 November.

The study

Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro et al.. Auditing AI peer reviewers: a benchmark on real scientific papers. arXiv, 12 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY 4.0

Notes

  1. Zubeldia, Bolliet, Villaescusa-Navarro, Villanueva-Domingo and Cranmer, 'Auditing AI peer reviewers: a benchmark on real scientific papers', arXiv:2611.17052, preprint, not peer reviewed. CC BY 4.0. Source
  2. Benchmark release: evaluation pipeline, paper lists, injection catalogues and adjudication logs. We have not checked the repository contents or licence. Source
  3. SKEPTHICAL public service. The paper does not state that its source code is released. Source
  4. Total detection, astrophysics: SKEPTHICAL 70.6%, structured GPT-5.5 61.3% (significant). ML: 76.7% against 81.1% (not significant). Free-form prompt, astrophysics: GPT-5.5 60.5%, Claude Opus 4.7 36.7%, Gemini 3.1 Pro 11.9%; ML: 63.9%, 43.9%, 11.1%.
  5. Citations, astrophysics: SKEPTHICAL 46.7%, next best 26.7%. ML: SKEPTHICAL 53.3%, best others 57.8 to 60.0%. Concise-prompt GPT-5.5: 68.9% (astrophysics), 76.1% (ML). False-positive rates for SKEPTHICAL: 2.8% and 4.0%.
  6. Cost per paper (astrophysics): SKEPTHICAL $2.41, about 24 minutes; structured single model about $0.55, about 2 minutes (Table 12).

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Safety & evaluation