# Arena-style AI leaderboards can be gamed with cheap, simple tricks

A preprint shows that votes, prompts and decoy models can lift a language model far up a crowd-voted ranking. The tests were simulations on public LMArena data, not attacks on the live site.

- Category: Safety & evaluation
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-15
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/arena-style-ai-leaderboards-can-be-gamed-with-cheap-simple-tricks

## The study

Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation (preprint, not peer reviewed). arXiv 2611.14769. https://arxiv.org/abs/2611.14769

## At a glance

- What it shows: Popular AI leaderboards rank models by asking people which of two anonymous answers they prefer. The authors show, in simulation, that a determined provider could push a model up such a ranking by 15 to 20 places with any one of three tricks.
- What it does not show: It does not show that anyone has actually rigged a live leaderboard. It also does not test the attacks on the largest LMArena leaderboard, which has about 50 times more votes.
- Why it matters: Companies use these rankings to choose models, and providers use them for publicity and funding. If the ranking can be bought cheaply, it is a weak basis for those decisions.

## What the study shows

Arena-style leaderboards work like this. A user types a question, two anonymous models answer, and the user picks the better answer. A statistical method called Bradley-Terry turns many such votes into a single ranking. The authors note that this method behaves like the Borda count, a points-based voting rule that political scientists have long known can be manipulated.

The authors tested three manipulations on a public set of about 100,000 votes from LMArena, covering 55 models. Strategic voting means casting votes that help a target model, including votes on matches it is not in. Strategic nomination means adding copies of an unrelated model that the target tends to beat. Strategic prompting means asking questions only in an area where the target does well, such as one language.

The authors report that each attack alone lifted some models by 15 to 20 places, using at most about 10% of the votes in the dataset. Typical gains were smaller. Prompting moved some models with as few as 500 added prompts. Combining nomination and prompting lifted one model by up to 45 places.

Voting attacks need the attacker to know which models are answering. The authors show this is often possible. Open-weight models, whose internals are public, were picked out with over 80% precision against a 25% chance rate. Closed models were harder to spot, at about 40 to 50%.

As a defence, the authors tested a reweighted ranking method borrowed from election design. It roughly halved the largest nomination boost. Other defences, such as bot checks and balancing prompts across topics, are proposed but not tested. This is a preprint and has not been peer reviewed.

## Our reading

In our reading, the most useful point is that this is a design flaw, not a bug in one site. The authors argue that any leaderboard using this ranking method and allowing open prompts, open voting or open model submission inherits the weakness. A leaderboard run entirely in good faith is still exposed.

The prompting attack is the most worrying of the three. Each vote is honest. The attacker only chooses which questions get asked. Consider a provider whose model is unusually strong in Hungarian. Its staff could simply ask many questions in Hungarian and vote truthfully, and the model would climb. That is hard to call cheating and hard to detect, especially if the chosen topic is less obvious than a language.

We rate this work at level II, Benchmarked, on our four-level readiness scale. The results come from a public dataset with clear comparisons. The code is promised but not yet released, so level III is not met. No leaderboard operator has adopted the defences, so level IV is not met either.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/arena-style-ai-leaderboards-can-be-gamed-with-cheap-simple-tricks

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
