At high temperature, the sampler decides whether long AI outputs survive
A preprint tests 29 open-weight models and finds that, once randomness is turned up, the rule that picks each next word can matter as much as the model itself. Coding agents went from about half of issues fixed to none on that one setting.


What the study shows
A language model writes one token at a time. A token is a word or part of a word. At each step the model gives a probability for every possible token, and a rule called the sampler picks one. Temperature is a dial for randomness: above 1, it spreads probability towards unlikely tokens.
Most samplers first cut away unlikely tokens, then pick from what remains. The authors call what remains the candidate set. The common default, top-p, keeps enough tokens to cover a fixed share of the probability, such as 90%. A newer rule, top-nσ, keeps only tokens whose raw scores sit close to the top score. Temperature cannot change which tokens top-nσ keeps.
The authors tested 29 open-weight models, from 1 billion to 122 billion parameters. They used ten public benchmarks plus a memory test out to one million tokens. Tasks covered coding agents, long step-by-step reasoning, long-form writing, long structured output and reading long documents.
At the default temperature of 1, the samplers scored about the same. On SWE-bench Verified, a set of real GitHub issues, every rule fixed roughly a third of 100 issues with a 9-billion-parameter Qwen model.
At temperature 2, agents using top-p, top-k or Top-H fixed none of 100 issues. Agents using top-nσ fixed between 46 and 62 out of 100 on fresh issue sets across three Qwen3.5 models. Across six issue grids, top-nσ won every issue where the two rules disagreed. The authors trace the failures to broken shell commands: one stray token makes a command unreadable.
The authors report the same pattern elsewhere, judged against each rule's own score at temperature 1. On long structured output scored by fixed rules, top-p roughly halved at temperature 1.5 while top-nσ held steady. In finding facts hidden in a million-token text, top-nσ retrieved about three quarters at 1.5 against about three fifths for top-p.
Their explanation is the size of the candidate set. In one example, moving from temperature 1 to 2 grew top-p's set from 4 tokens to 51,845. Top-nσ stayed at 3. Across 133 combinations of setting and temperature, larger typical sets went with lower judged quality.
To test cause, they picked the candidate set at temperature 1 and only then applied temperature. Three rules that had collapsed, Top-H, min-p and P-less, then kept their quality up to temperature 3. This is a preprint and has not been peer reviewed.
Our reading
In our reading, the most useful finding is not that one sampler wins. It is that a setting most teams never touch can turn a working agent into a broken one. The collapse in coding agents is total and consistent: zero issues fixed, not a gradual drop.
The causal test is the strongest part. Freezing the candidate set before adding randomness rescued three rules that had collapsed. That points to the unlikely tail of tokens, not randomness as such, as the source of harm. It also gives teams a second fix besides switching rules.
The claims are narrower than the title suggests. Every result that separates the samplers comes from raised temperature, between 1.3 and 2. At the default setting they tie, apart from a small lead on one long-reading benchmark. The study does not show that raising temperature is worth doing at all. Top-nσ mostly stays flat rather than improving.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | Controlled experiments on open-weight models with a logits processor that replaces native sampling. |
| II Benchmarked | Met | Results on public benchmarks: SWE-bench Verified, LiveCodeBench v6, AIME/HMMT, LongBench-Write, WritingBench, LongProc, LongBench v2 and needle retrieval, with paired tests against top-p. |
| III Open source | Not yet | The abstract says all code and data are released, but the supplied text gives no URL, so we cannot verify it. The samplers themselves are existing methods, and top-nσ is stated to be in llama.cpp. Treat this level as likely but unconfirmed until the repository link is checked. |
| IV In production | Not yet | No named organisation is documented as using the sampler setting in production. The agent runs are experiments, not deployments. |
What would move it up
Confirm the released code and data at a public link (which would reach level 3). For level 4, a named team would need to report running a tail-excluding sampler such as top-nσ in a live coding agent or long-output service and compare it with their previous setup. Independent replication and human evaluation of writing quality would also help.
The study
Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv et al.. Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat). arXiv, 17 November 2026. Read the paper
Notes
- Roush, Nguyen, Shwartz-Ziv, Goldfeder and Basu, 'Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)', arXiv preprint, 17 November 2026. Marked for the NeurIPS 2026 Evaluations and Datasets track. CC BY 4.0. Source
- The abstract says all code and data are released, but no repository link appears in the text we read.
- Coding agents: at temperature 2 on 100 fresh SWE-bench Verified issues, top-nσ resolved 46%, 62% and 53% (Qwen3.5-9B, 27B, 35B-A3B); top-p and Top-H resolved 0%. Across six grids top-nσ won 277 disputed issues to 0 (Figure 11, Tables 3 to 5).
- Candidate-set sizes, median per step, temperature 1 to 2: top-p 2 to about 90,000; Top-H 1 to 54; P-less 1 to 21; min-p 1 to 14; top-nσ 1 to 1 (Table 1, Table 15). Llama-3.1-8B example: top-p 4 to 51,845, top-nσ 3 (Figure 1a).
- LongProc at temperature 1.5: top-nσ 28.5% (29.3% at T=1), top-p 13.4% (29.1% at T=1) (Table 9). Needle retrieval at one million tokens, T=1.5: top-nσ 75%, top-p 59% (Section 5.3). Set size against judged quality: Spearman -0.77 over 133 cells (Figure 7).
- Temperature-last test: Top-H, min-p and P-less keep 4.1 to 4.6 out of 10 up to temperature 3, against 1.1 to 1.2 in the default order (Section 5.7, Table 16).
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections
