Some open base models recite benchmark test questions they were never shown
A preprint finds that two of four recent open base models write out benchmark test items, answers included, from memory. Zeroing those items drops one model from first to last.

AI illustrationWhat the study shows
The authors studied benchleaking. This is their term for a benchmark's own test examples ending up in a model's training data, so that a score reflects recall rather than skill. They separate it from benchmaxxing, where developers tune hard towards a benchmark without seeing its test set.
They tested four public base models. A base model is the raw pre-trained model, before it is tuned to follow instructions. The models were ERNIE-5.5-Flash-Base, MiMo-V2.5-Base, DeepSeek-V4-Flash-Base and Hy3-preview-Base, across 20 public benchmarks covering knowledge, reasoning, maths, code and science.
The probe was simple. Each model saw a few solved examples and one test question. The authors then let it keep writing past its answer. They checked whether it produced other items from the same test set. An item only counted as leaked if the full question, all options and the reference answer matched, ignoring only case and spacing.
The authors report that ERNIE-5.5-Flash-Base reproduced about 96% of HumanEval, a coding test, with its reference code, and about 85% of OpenBookQA, a science quiz. In one case it copied a spelling mistake from the original test file. MiMo-V2.5-Base reproduced about three quarters of HumanEval. The other two models leaked under a tenth of HumanEval and nothing from OpenBookQA.
The authors then define an Anti-Hack score. It is the normal benchmark score with every detected leaked item marked wrong. Averaged over the 20 benchmarks, DeepSeek and Hy3 lose under 2 points each. MiMo loses 5. ERNIE loses about 17 and falls from the highest average of the four to the lowest.
A smaller part of the paper looks at agents, which are models that carry out multi-step tasks with tools. The authors say they repeated two earlier reports, neither peer reviewed. In one, an agent called Kimi-K3.1-Pro stated grading rules from hidden tests that its task instructions never mentioned. A judge model flagged this on 9 of 63 tasks, against 1 for its predecessor.
The paper is a preprint and has not been peer reviewed.
Our reading
In our reading, the base-model evidence is the strong part of this paper. A model that writes out the next four HumanEval problems in order, typo included, is hard to explain without exposure to the test set. The strict matching rule, which rejects paraphrases and code that merely works, makes false alarms unlikely.
The practical consequence is concrete. Imagine a team picking a base model to fine-tune for code. On the reported HumanEval scores, ERNIE-5.5-Flash-Base scores a perfect 100 and looks like the obvious choice. With leaked items zeroed, it scores under 4. The leaderboard would have pointed the team at the wrong model.
The headline claim that the problem is accelerating is weaker. Each model was examined once. There is no time series, so the paper cannot show a trend over the past month. The agent findings are also softer than the base-model ones: they rest on small task counts and an automatic judge whose accuracy is not reported in the main text.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The method and results are described in full as a research preprint, with worked examples (Figures 1, 3, 5 to 7). |
| II Benchmarked | Met | Results are reported on 20 public benchmarks (including HumanEval, MMLU, GSM8K, GPQA-Main) for four public base models, with scores before and after adjustment in Tables 4 and 7 to 9. |
| III Open source | Met | The paper links a public repository at https://github.com/GAIR-NLP/Benchleaking. The paper does not say what it contains or its licence, so the contents are unchecked. |
| IV In production | Not yet | No named organisation is documented as using the Anti-Hack score or the probe in production. Adoption is only called for in the conclusion. |
What would move it up
Level 4 needs documented use by a named organisation, such as a model developer or leaderboard adopting contamination-adjusted scores in routine reporting. Independent replication on more models, and confirmation from model makers about training data, would also strengthen the findings.
The study
Yiwei Qin, Muhang Xie, Xuefeng Li et al.. Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models. arXiv, 16 November 2026. Read the paper
Notes
- Qin, Xie, Li, Jiang, Fu, Mi, Guo and Liu. Benchleaking Is Accelerating. arXiv preprint 2611.21315, 16 November 2026. Not peer reviewed. CC BY 4.0. Source
- Code repository linked from the paper. Contents and licence not checked by us. Source
- Leakage under the strictest matching rule (Table 4): ERNIE-5.5-Flash-Base HumanEval 96.3%, OpenBookQA 85.4%, GSM8K 80.1%, GPQA-Main 34.6%. MiMo-V2.5-Base HumanEval 73.8%. DeepSeek-V4-Flash-Base HumanEval 9.1%. Hy3-preview-Base HumanEval 6.7%.
- 20-benchmark averages, original to Anti-Hack (Table 9): DeepSeek-V4-Flash-Base 77.95 to 76.48; Hy3-preview-Base 79.86 to 78.55; MiMo-V2.5-Base 80.01 to 75.03; ERNIE-5.5-Flash-Base 85.17 to 68.55.
- ERNIE-5.5-Flash-Base single benchmarks, original to Anti-Hack: HumanEval 100.00 to 3.66; OpenBookQA 98.60 to 14.20; GSM8K 91.21 to 17.21.
- Agent results (Table 3): Kimi-K3.1-Pro stated hidden-rule candidates on 9 of 63 tasks while solving (14.3%) against 1 of 63 for Kimi-K3 (1.6%); under prefilled reasoning, 34 of 66 (51.5%) against 6 of 66 (9.1%). MaxxBench pass@1 drops (Table 2): DeepSeek-V4.1-Flash 75.8% to 62.5%; Kimi-K3.1-Pro 83.8% to 68.0%.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

