# At high temperature, the sampler decides whether long AI outputs survive

A preprint tests 29 open-weight models and finds that, once randomness is turned up, the rule that picks each next word can matter as much as the model itself. Coding agents went from about half of issues fixed to none on that one setting.

- Category: Language models
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-18
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/at-high-temperature-the-sampler-decides-whether-long-ai-outputs-survive

## The study

Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat) (preprint, not peer reviewed). arXiv 2611.22148. https://arxiv.org/abs/2611.22148

## At a glance

- What it shows: When language models are set to be more random, the rule that picks each next word decides whether long outputs stay usable. Some rules held steady, while the most common one collapsed, including in coding agents that went from fixing about half of test issues to none.
- What it does not show: It does not show that more randomness helps, or that any rule fixes long writing in general. At the normal setting, the rules perform about the same.
- Why it matters: Teams running open models can avoid a whole class of failures by changing a free setting rather than retraining. It also means benchmark scores can swing widely on a setting that is rarely reported.

## What the study shows

A language model writes one token at a time. A token is a word or part of a word. At each step the model gives a probability for every possible token, and a rule called the sampler picks one. Temperature is a dial for randomness: above 1, it spreads probability towards unlikely tokens.

Most samplers first cut away unlikely tokens, then pick from what remains. The authors call what remains the candidate set. The common default, top-p, keeps enough tokens to cover a fixed share of the probability, such as 90%. A newer rule, top-nσ, keeps only tokens whose raw scores sit close to the top score. Temperature cannot change which tokens top-nσ keeps.

The authors tested 29 open-weight models, from 1 billion to 122 billion parameters. They used ten public benchmarks plus a memory test out to one million tokens. Tasks covered coding agents, long step-by-step reasoning, long-form writing, long structured output and reading long documents.

At the default temperature of 1, the samplers scored about the same. On SWE-bench Verified, a set of real GitHub issues, every rule fixed roughly a third of 100 issues with a 9-billion-parameter Qwen model.

At temperature 2, agents using top-p, top-k or Top-H fixed none of 100 issues. Agents using top-nσ fixed between 46 and 62 out of 100 on fresh issue sets across three Qwen3.5 models. Across six issue grids, top-nσ won every issue where the two rules disagreed. The authors trace the failures to broken shell commands: one stray token makes a command unreadable.

The authors report the same pattern elsewhere, judged against each rule's own score at temperature 1. On long structured output scored by fixed rules, top-p roughly halved at temperature 1.5 while top-nσ held steady. In finding facts hidden in a million-token text, top-nσ retrieved about three quarters at 1.5 against about three fifths for top-p.

Their explanation is the size of the candidate set. In one example, moving from temperature 1 to 2 grew top-p's set from 4 tokens to 51,845. Top-nσ stayed at 3. Across 133 combinations of setting and temperature, larger typical sets went with lower judged quality.

To test cause, they picked the candidate set at temperature 1 and only then applied temperature. Three rules that had collapsed, Top-H, min-p and P-less, then kept their quality up to temperature 3. This is a preprint and has not been peer reviewed.

## Our reading

In our reading, the most useful finding is not that one sampler wins. It is that a setting most teams never touch can turn a working agent into a broken one. The collapse in coding agents is total and consistent: zero issues fixed, not a gradual drop.

The causal test is the strongest part. Freezing the candidate set before adding randomness rescued three rules that had collapsed. That points to the unlikely tail of tokens, not randomness as such, as the source of harm. It also gives teams a second fix besides switching rules.

The claims are narrower than the title suggests. Every result that separates the samplers comes from raised temperature, between 1.3 and 2. At the default setting they tie, apart from a small lead on one long-reading benchmark. The study does not show that raising temperature is worth doing at all. Top-nσ mostly stays flat rather than improving.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/at-high-temperature-the-sampler-decides-whether-long-ai-outputs-survive

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
