# Some open base models recite benchmark test questions they were never shown

A preprint finds that two of four recent open base models write out benchmark test items, answers included, from memory. Zeroing those items drops one model from first to last.

- Category: Safety & evaluation
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-17
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/some-open-base-models-recite-benchmark-test-questions-they-were-never-shown

## The study

Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models (preprint, not peer reviewed). arXiv 2611.21315. https://arxiv.org/abs/2611.21315

## At a glance

- What it shows: When given a few solved examples and one test question, some open AI models carried on and wrote out other test questions with the correct answers. Those items were not in what they were given. One model did this for almost every problem in a popular coding test.
- What it does not show: It does not show how the test data got into training, or whether it was deliberate. It also does not prove that models with little detected leakage are clean.
- Why it matters: Benchmark scores guide which models companies adopt and build on. If a model has memorised the test, its score says little about how it will perform on real work.

## What the study shows

The authors studied benchleaking. This is their term for a benchmark's own test examples ending up in a model's training data, so that a score reflects recall rather than skill. They separate it from benchmaxxing, where developers tune hard towards a benchmark without seeing its test set.

They tested four public base models. A base model is the raw pre-trained model, before it is tuned to follow instructions. The models were ERNIE-5.5-Flash-Base, MiMo-V2.5-Base, DeepSeek-V4-Flash-Base and Hy3-preview-Base, across 20 public benchmarks covering knowledge, reasoning, maths, code and science.

The probe was simple. Each model saw a few solved examples and one test question. The authors then let it keep writing past its answer. They checked whether it produced other items from the same test set. An item only counted as leaked if the full question, all options and the reference answer matched, ignoring only case and spacing.

The authors report that ERNIE-5.5-Flash-Base reproduced about 96% of HumanEval, a coding test, with its reference code, and about 85% of OpenBookQA, a science quiz. In one case it copied a spelling mistake from the original test file. MiMo-V2.5-Base reproduced about three quarters of HumanEval. The other two models leaked under a tenth of HumanEval and nothing from OpenBookQA.

The authors then define an Anti-Hack score. It is the normal benchmark score with every detected leaked item marked wrong. Averaged over the 20 benchmarks, DeepSeek and Hy3 lose under 2 points each. MiMo loses 5. ERNIE loses about 17 and falls from the highest average of the four to the lowest.

A smaller part of the paper looks at agents, which are models that carry out multi-step tasks with tools. The authors say they repeated two earlier reports, neither peer reviewed. In one, an agent called Kimi-K3.1-Pro stated grading rules from hidden tests that its task instructions never mentioned. A judge model flagged this on 9 of 63 tasks, against 1 for its predecessor.

The paper is a preprint and has not been peer reviewed.

## Our reading

In our reading, the base-model evidence is the strong part of this paper. A model that writes out the next four HumanEval problems in order, typo included, is hard to explain without exposure to the test set. The strict matching rule, which rejects paraphrases and code that merely works, makes false alarms unlikely.

The practical consequence is concrete. Imagine a team picking a base model to fine-tune for code. On the reported HumanEval scores, ERNIE-5.5-Flash-Base scores a perfect 100 and looks like the obvious choice. With leaked items zeroed, it scores under 4. The leaderboard would have pointed the team at the wrong model.

The headline claim that the problem is accelerating is weaker. Each model was examined once. There is no time series, so the paper cannot show a trend over the past month. The agent findings are also softer than the base-model ones: they rest on small task counts and an automatic judge whose accuracy is not reported in the main text.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/some-open-base-models-recite-benchmark-test-questions-they-were-never-shown

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
