# People judged unchanged chatbots more harshly over 16 months, study finds

An analysis of 1.26 million blind votes on Chatbot Arena finds that verdicts on fixed AI models grew harsher while rankings held steady. The authors argue the bar moved, not the models. The paper is a preprint and has not been peer reviewed.

- Category: Safety & evaluation
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-12
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/people-judged-unchanged-chatbots-more-harshly-over-16-months-study-finds

## The study

AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.14333, arXiv 2611.14333. https://arxiv.org/abs/2611.14333

## At a glance

- What it shows: On Chatbot Arena, the share of votes rejecting both answers from the same pair of unchanged models rose by 1.71 percentage points a month between May 2023 and August 2024. Over the same period, which model voters preferred did not shift in 2024.
- What it does not show: It does not show why voters became harsher, because no one recorded what other AI systems each voter had used. It also covers one self-selected platform and stops in August 2024.
- Why it matters: Providers, buyers and model trainers read falling human ratings as evidence about the system. If the raters' standard drifts, a stable product can look like it is getting worse, and feedback labels from different periods may mean different things.

## What the study shows

Chatbot Arena is a public website where a visitor types a prompt, sees answers from two anonymous large language models (LLMs) side by side, and votes: left is better, right is better, tie, or both are bad. Model names appear only after the vote. The authors used the public vote log that the platform's operator, LMSYS, released in August 2024.

They kept only battles between two "frozen" models: dated vendor snapshots such as gpt-4-0613, whose version is published as fixed, and open-weight models released as a single checkpoint. That left 1,260,967 votes on 2,231 model pairs, cast from 1 May 2023 to 14 August 2024. Names that can point to a changing model, such as undated aliases, were excluded.

The main measure is how often voters chose "both are bad". Comparing each pair only with itself over time, the authors report that this share rose by 1.71 percentage points a month (standard error 0.18) on an average of 17.1%. In 2024 alone, when 92% of these votes were cast, it rose by 1.91 points a month.

A simpler check uses the fourteen pairs with at least 50 votes in both May to October 2023 and February to August 2024. The share rose in all fourteen, from 15.5% to 23.3% on a weighted basis. These include gpt-4-0613 against claude-2.0, among the strongest pairs of 2023, as well as small open models.

The extra rejections came out of decisive votes, not ties. Ties barely moved (minus 0.09 points a month). Among decisive votes, the stronger model's share of wins showed no trend in 2024 (0.00 points a month, standard error 0.13). The authors read this as voters becoming stricter about what counts as acceptable, while still picking the better answer just as reliably.

The authors tested other explanations. The rise stayed positive across 28 variants of the analysis, from 0.97 to 2.61 points a month. These included controls for prompt language and difficulty tags, easy single-turn English prompts only (1.30), each of the seven most common languages, pairs of dated OpenAI and Anthropic snapshots only (1.41), open-weight pairs only (2.12), models whose answer length stayed stable (1.16), and separate periods between changes to the Arena web page.

The rise also appeared within the same voter. Voters are tracked by a hashed identifier derived from their network address. Among 24,508 identifiers seen over at least a month, the share rose by 1.90 points a month. For the same identifier judging the same pair more than a month apart, it rose by 2.30, though on only 11,642 votes. Newcomers were harsher too: each identifier's first recorded vote became 1.25 points more likely to be "both are bad" for each month later that they arrived.

Seeing a strong model in the previous battle did not make the next verdict harsher. The authors take this as a sign that the standard moves slowly with accumulated experience, not in response to a single encounter.

Finally, the drift was shared. Seventeen of eighteen heavily judged frozen models showed a significant rise on their own pairs, averaging 1.40 points a month. When the authors subtracted monthly effects estimated only from other models' pairs, the average fell to 0.19. Platform-wide, the average model in battles gained 245 points on the Elo scale, a chess-style rating, yet the raw share of "both are bad" votes was 19.4% in May 2023 and 20.6% in August 2024.

## Our reading

In our reading, this is a careful piece of measurement on a question that usually runs on anecdote. The design is the strength: the same fixed models, judged blind, over more than a year, with the same voters followed where possible. The authors work hard to rule out the obvious alternatives, and the result survives each test they can run with the data available.

The split between absolute and relative verdicts is the most useful finding for practitioners. Asking "which answer is better?" gave stable results in 2024. Asking, in effect, "is either answer good enough?" did not. Any system that relies on human acceptability judgements over time, from satisfaction scores to training labels, should treat them as tied to the date they were collected.

The explanation the authors favour, that standards rise as the whole category of AI improves, fits the data but is not directly tested. Nobody recorded what other assistants each voter used, and average model quality on Arena rose almost in a straight line with time, so it cannot be separated from anything else that changed steadily over those months. The study shows harsher judges; it does not show exactly why.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/people-judged-unchanged-chatbots-more-harshly-over-16-months-study-finds

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
