A training loss that targets what speculative decoding actually rewards
A single-author preprint argues that popular losses for training draft models measure the wrong thing, and proposes an exact alternative. The gains on one 35B model are consistent but small.

AI illustrationWhat the study shows
Speculative decoding speeds up a large language model by letting a small 'draft' model guess several tokens ahead. The large 'target' model checks the guesses in one pass and keeps them up to the first rejection. The speedup depends on the expected acceptance length (EAL): how many draft tokens survive per round on average. This paper is a preprint and has not been peer reviewed.
Draft models are usually trained to imitate the target token by token, for example with cross-entropy (a standard prediction loss). Newer losses try to target EAL directly. One family, including windowed total variation (WTV), multiplies the average acceptance rate at each step. The author argues this ignores which tokens actually got accepted, because an accepted token sets the context for checking the next one. The missing piece is a covariance (a measure of how two quantities move together) between how likely a token is to be accepted and how easy the next step becomes.
The author gives a hand-built two-step example where WTV and the true objective have opposite gradients (+0.20 against −0.12) at the same point, so the surrogate would steer training away from better acceptance. A second toy example shows WTV ranking two draft models the wrong way round.
The proposed loss, REAL, estimates the true EAL on text sampled from the target. Each sampled token gets a weight that corrects for how the target and draft differ. To reduce noise, the final step of each term is replaced by its exact expected value, a standard statistical technique called Rao-Blackwellization. A small cross-entropy term (weight 0.1) is kept to stabilise training. Nothing else changes: not the drafter design, not the inference procedure.
The main test used a frozen Qwen3.6-35B-A3B target and a Domino drafter, trained for 3 epochs (about 140,000 steps) on prompts from the Open-PerfectBlend dataset with answers regenerated by the target. Evaluation ran in the vLLM serving engine on 9 task sets, including HumanEval, GSM8K, MATH-500 and BFCL, with 80 requests each and 5 seeds.
Under probabilistic serving at temperature 1, REAL averaged 4.792 accepted tokens per round. The author's reimplementations of four rival losses scored 4.657 (VSD), 4.715 (WTV), 4.740 (static LK) and 4.749 (DBLast). REAL or its variant without Rao-Blackwellization (4.791) was best on every task set. Under greedy serving, where the draft always proposes its top token, static LK was best at 4.544 against 4.537 for REAL. At temperature 0.5, static LK also edged ahead (5.002 against 4.993).
Rao-Blackwellization cut aggregate gradient variance by 52 to 58%. It made no difference to final acceptance length at temperature 1 but held up better at lower temperatures. Removing the cross-entropy term (4.624) or using cross-entropy alone (4.668) both did worse than REAL. Serving throughput was 4,662 tokens per second for REAL against 4,649 for DBLast and 4,553 for WTV. Training step times were similar across methods, at roughly 835 to 850 milliseconds.
Our reading
In our reading, the strongest part of this paper is the diagnosis, not the benchmark. The argument that multiplying per-step acceptance rates is not the same as the real objective is simple, checkable and backed by worked examples. Anyone training draft models with a WTV-style loss has reason to look again at what that loss rewards. The author also shows that a WTV-shaped training metric favoured WTV while serving results favoured REAL, which is a useful warning about trusting training-time proxies.
The benchmark gains are real but modest. REAL beats the strongest rival by 0.043 accepted tokens per round, under 1%. The spread across seeds reaches 0.021, and no confidence intervals are reported. The consistency is encouraging: REAL led on every seed and every task set under sampled serving. But all baselines were reimplemented by the author at the loss level, with no described tuning, so the true gap to well-tuned rivals is unknown.
On readiness, we place this at level III, open source available. Results on public benchmarks meet level II, and a reference implementation of the loss is released as a branch of the Red Hat Speculators library. We have not checked the repository's licence or completeness, and no trained weights are mentioned. There is no documented production use.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The method is derived and tested with a clear lab-style setup: one 35B MoE target, one drafter, and a small pilot. |
| II Benchmarked | Met | Results on 9 public task datasets (HumanEval, GSM8K, MATH500, MBPP, NQ, BFCL and others), 5 seeds, against 5 baselines. REAL leads on probabilistic acceptance length (4.792 vs 4.657-4.749). Baselines are the author's own loss-level reimplementations, and the margins are small. |
| III Open source | Met | A reference implementation of the loss is released at github.com/King-ty/speculators/tree/real-loss, a branch of the Red Hat/vLLM Speculators library. The paper does not state a licence or release trained weights. I could not check the repository. |
| IV In production | Not yet | No named organisation is reported using REAL in production. The paper's evidence is one offline vLLM benchmark with 80 requests per subset. |
What would move it up
To reach level 4, a named team would need to report deploying REAL-trained drafters in live serving, with measured latency or cost gains over its existing drafter. Before that, replication on other targets, drafters and datasets would help, as would baselines run with their authors' own code and tuned settings, confidence intervals on the small margins, and end-to-end speedup against non-speculative decoding.
Checked automatically every week, last on 16 November.
The study
Yu Tian. REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding. arXiv, 10 November 2026. doi:10.48550/arxiv.2611.14399
Notes
- Tian, Y. REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding. arXiv preprint 2611.14399, November 2026. Not peer reviewed. Source
- Reference implementation of the REAL loss, a branch of the Speculators library. Licence and contents not checked by us. Source
- Speculators, the Red Hat library for training speculative decoding drafters used in the study. Source
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

