Arena-style AI leaderboards can be gamed with cheap, simple tricks

A preprint shows that votes, prompts and decoy models can lift a language model far up a crowd-voted ranking. The tests were simulations on public LMArena data, not attacks on the live site.

5 min readIIBenchmarkedPreprint, not peer reviewed
Save
Figure 4 from the paper "Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation"
Rank gains for each model when decoy model copies are added and prompts are then steered towards the target's strongest language, with one model rising 45 places and most rising far less. Figure 4 from Eve Fleisig et al., 2026, arXiv:2611.14769. CC BY 4.0.

What the study shows

Arena-style leaderboards work like this. A user types a question, two anonymous models answer, and the user picks the better answer. A statistical method called Bradley-Terry turns many such votes into a single ranking. The authors note that this method behaves like the Borda count, a points-based voting rule that political scientists have long known can be manipulated.

The authors tested three manipulations on a public set of about 100,000 votes from LMArena, covering 55 models. Strategic voting means casting votes that help a target model, including votes on matches it is not in. Strategic nomination means adding copies of an unrelated model that the target tends to beat. Strategic prompting means asking questions only in an area where the target does well, such as one language.

The authors report that each attack alone lifted some models by 15 to 20 places, using at most about 10% of the votes in the dataset. Typical gains were smaller. Prompting moved some models with as few as 500 added prompts. Combining nomination and prompting lifted one model by up to 45 places.

Voting attacks need the attacker to know which models are answering. The authors show this is often possible. Open-weight models, whose internals are public, were picked out with over 80% precision against a 25% chance rate. Closed models were harder to spot, at about 40 to 50%.

As a defence, the authors tested a reweighted ranking method borrowed from election design. It roughly halved the largest nomination boost. Other defences, such as bot checks and balancing prompts across topics, are proposed but not tested. This is a preprint and has not been peer reviewed.

Our reading

In our reading, the most useful point is that this is a design flaw, not a bug in one site. The authors argue that any leaderboard using this ranking method and allowing open prompts, open voting or open model submission inherits the weakness. A leaderboard run entirely in good faith is still exposed.

The prompting attack is the most worrying of the three. Each vote is honest. The attacker only chooses which questions get asked. Consider a provider whose model is unusually strong in Hungarian. Its staff could simply ask many questions in Hungarian and vote truthfully, and the model would climb. That is hard to call cheating and hard to detect, especially if the chosen topic is less obvious than a language.

We rate this work at level II, Benchmarked, on our four-level readiness scale. The results come from a public dataset with clear comparisons. The code is promised but not yet released, so level III is not met. No leaderboard operator has adopted the defences, so level IV is not met either.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

II
Benchmarked
LevelStatusEvidence
I  Lab onlyMetThe attacks and the harmonic defence are demonstrated in simulation on a public dataset.
II  BenchmarkedMetResults come from a public dataset (LMArena arena-human-preference-100k, CC-BY-4.0), with comparisons against the original rankings and a random-removal baseline. This is an evaluation of attacks rather than of a model, so level 2 fits loosely.
III  Open sourceNot yetThe paper says code will be released with the camera-ready version. No link is given. Only the input data is public.
IV  In productionNot yetNo named organisation has adopted the defences. The attacks were not tested on live leaderboards.

What would move it up
Releasing the attack and defence code would reach level 3. Level 4 would need a leaderboard operator to adopt the defences and report the effect on live rankings. Testing the attacks on larger arenas, with the operator's agreement, would also strengthen the case.

Checked automatically every week, last on 16 November.

The study

Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell et al.. Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation. arXiv, 10 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY 4.0

Notes

  1. Fleisig, Baumann, Hadfield-Menell and Hovy. Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation. arXiv preprint, November 2026, not peer reviewed. Marked for NeurIPS 2026. Source
  2. Data used: LMArena arena-human-preference-100k, CC BY 4.0. About 100,000 votes covering 55 models. Source
  3. Exact figures: greedy voting up to 20 places (23 models over 10 places); heuristic voting up to 15 places (21 models over 10 places, 15 into top 5); nomination up to 15 places with 10 copies (typical 1 to 7); prompting up to 15 places (up to 14 with topic labels); combined up to 45 places (9 models over 10 places). Vote budgets capped at 10,000, about 10% of the data.
  4. Model identification: over 80% precision for open-weight models (except qwen2-72b-instruct), about 40 to 50% for three GPT models, against a 25% base rate.
  5. Harmonic defence at beta 0.75: maximum nomination boost cut from 15 to 8 places; mean boost 2.74 to 2.81; Kendall tau with original ranking 0.946.
  6. Code: the authors say it will be released with the camera-ready version. No link at time of writing.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Safety & evaluation