People judged unchanged chatbots more harshly over 16 months, study finds

An analysis of 1.26 million blind votes on Chatbot Arena finds that verdicts on fixed AI models grew harsher while rankings held steady. The authors argue the bar moved, not the models. The paper is a preprint and has not been peer reviewed.

8 min readIIBenchmarkedPreprint, not peer reviewed
Save
Figure 1 from the paper "AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models"
Figure 1, panel (b), or Figure 3: the same frozen model pairs show a steady rise in 'both are bad' votes while the raw platform-wide share stays flat (Figure 1b), and the stronger model's win share is flat (Figure 3b). The paper is CC BY, so reuse is allowed with credit. Figure 1 from Zihan Chen et al., 2026, arXiv:2611.14333. CC BY.

What the study shows

Chatbot Arena is a public website where a visitor types a prompt, sees answers from two anonymous large language models (LLMs) side by side, and votes: left is better, right is better, tie, or both are bad. Model names appear only after the vote. The authors used the public vote log that the platform's operator, LMSYS, released in August 2024.

They kept only battles between two "frozen" models: dated vendor snapshots such as gpt-4-0613, whose version is published as fixed, and open-weight models released as a single checkpoint. That left 1,260,967 votes on 2,231 model pairs, cast from 1 May 2023 to 14 August 2024. Names that can point to a changing model, such as undated aliases, were excluded.

The main measure is how often voters chose "both are bad". Comparing each pair only with itself over time, the authors report that this share rose by 1.71 percentage points a month (standard error 0.18) on an average of 17.1%. In 2024 alone, when 92% of these votes were cast, it rose by 1.91 points a month.

A simpler check uses the fourteen pairs with at least 50 votes in both May to October 2023 and February to August 2024. The share rose in all fourteen, from 15.5% to 23.3% on a weighted basis. These include gpt-4-0613 against claude-2.0, among the strongest pairs of 2023, as well as small open models.

The extra rejections came out of decisive votes, not ties. Ties barely moved (minus 0.09 points a month). Among decisive votes, the stronger model's share of wins showed no trend in 2024 (0.00 points a month, standard error 0.13). The authors read this as voters becoming stricter about what counts as acceptable, while still picking the better answer just as reliably.

The authors tested other explanations. The rise stayed positive across 28 variants of the analysis, from 0.97 to 2.61 points a month. These included controls for prompt language and difficulty tags, easy single-turn English prompts only (1.30), each of the seven most common languages, pairs of dated OpenAI and Anthropic snapshots only (1.41), open-weight pairs only (2.12), models whose answer length stayed stable (1.16), and separate periods between changes to the Arena web page.

The rise also appeared within the same voter. Voters are tracked by a hashed identifier derived from their network address. Among 24,508 identifiers seen over at least a month, the share rose by 1.90 points a month. For the same identifier judging the same pair more than a month apart, it rose by 2.30, though on only 11,642 votes. Newcomers were harsher too: each identifier's first recorded vote became 1.25 points more likely to be "both are bad" for each month later that they arrived.

Seeing a strong model in the previous battle did not make the next verdict harsher. The authors take this as a sign that the standard moves slowly with accumulated experience, not in response to a single encounter.

Finally, the drift was shared. Seventeen of eighteen heavily judged frozen models showed a significant rise on their own pairs, averaging 1.40 points a month. When the authors subtracted monthly effects estimated only from other models' pairs, the average fell to 0.19. Platform-wide, the average model in battles gained 245 points on the Elo scale, a chess-style rating, yet the raw share of "both are bad" votes was 19.4% in May 2023 and 20.6% in August 2024.

Our reading

In our reading, this is a careful piece of measurement on a question that usually runs on anecdote. The design is the strength: the same fixed models, judged blind, over more than a year, with the same voters followed where possible. The authors work hard to rule out the obvious alternatives, and the result survives each test they can run with the data available.

The split between absolute and relative verdicts is the most useful finding for practitioners. Asking "which answer is better?" gave stable results in 2024. Asking, in effect, "is either answer good enough?" did not. Any system that relies on human acceptability judgements over time, from satisfaction scores to training labels, should treat them as tied to the date they were collected.

The explanation the authors favour, that standards rise as the whole category of AI improves, fits the data but is not directly tested. Nobody recorded what other assistants each voter used, and average model quality on Arena rose almost in a straight line with time, so it cannot be separated from anything else that changed steadily over those months. The study shows harsher judges; it does not show exactly why.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

II
Benchmarked
LevelStatusEvidence
I  Lab onlyMetThe work is a research analysis with a testable claim, and it goes beyond lab-only status because it uses real public data.
II  BenchmarkedMetThe claims rest on a large public dataset (1.26 million votes from the Chatbot Arena battle log, with the file checksum given) and many robustness checks, including 28 specifications and an out-of-sample re-anchoring test. This is an empirical analysis, not a model benchmark, so Level II applies only loosely.
III  Open sourceNot yetThe paper links only the LMSYS public battle log it analyses. It does not release analysis code, a re-anchoring tool or derived data. No repository is named.
IV  In productionNot yetThere is no documented use of the re-anchoring approach by a named organisation. The suggestion that platforms keep frozen reference models is a recommendation, not a deployment.

What would move it up
Releasing the analysis code and derived tables would reach Level III. A named evaluation platform or AI provider adopting frozen reference items to re-anchor absolute verdicts would reach Level IV. Independent replication on other platforms or on later data, and a controlled experiment on rater exposure, would strengthen the causal claim.

Checked automatically every week, last on 16 November.

The study

Zihan Chen, Lei Nico Zheng, Di Zhu. AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models. arXiv, 10 November 2026. doi:10.48550/arxiv.2611.14333

Preprint, not peer reviewed · Licence CC BY · Stevens Institute of Technology, Hoboken

Notes

  1. Chen, Zheng and Zhu, "AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models", arXiv preprint, November 2026, not peer reviewed. CC BY licence. Source
  2. The public Chatbot Arena battle log analysed in the study (LMSYS release of 14 August 2024). This is the input data, not code from the authors. No analysis code has been released. Source
  3. FastChat, the open-source software that runs Chatbot Arena, whose commit history the authors used to date interface changes. Source

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Safety & evaluation