# A training loss that targets what speculative decoding actually rewards

A single-author preprint argues that popular losses for training draft models measure the wrong thing, and proposes an exact alternative. The gains on one 35B model are consistent but small.

- Category: Efficiency
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-12
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/a-training-loss-that-targets-what-speculative-decoding-actually-rewards

## The study

REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.14399, arXiv 2611.14399. https://arxiv.org/abs/2611.14399

## At a glance

- What it shows: Changing only the training loss of a draft model raised average acceptance length from 4.749 (best baseline) to 4.792 on a Qwen3.6-35B-A3B target across 9 task sets and 5 seeds, under sampled (non-greedy) serving. The author also shows mathematically that a widely used surrogate loss can push training in the wrong direction.
- What it does not show: It does not show the gain holds for other models, drafters or data, or that it survives greedy or low-temperature serving, where an older loss does as well or better. It reports no speedup against running without speculation and no significance tests on small margins.
- Why it matters: Speculative decoding is a standard way to cut inference cost. A loss that lines training up with the serving metric, at no extra training cost, is easy to adopt if the result replicates.

## What the study shows

Speculative decoding speeds up a large language model by letting a small 'draft' model guess several tokens ahead. The large 'target' model checks the guesses in one pass and keeps them up to the first rejection. The speedup depends on the expected acceptance length (EAL): how many draft tokens survive per round on average. This paper is a preprint and has not been peer reviewed.

Draft models are usually trained to imitate the target token by token, for example with cross-entropy (a standard prediction loss). Newer losses try to target EAL directly. One family, including windowed total variation (WTV), multiplies the average acceptance rate at each step. The author argues this ignores which tokens actually got accepted, because an accepted token sets the context for checking the next one. The missing piece is a covariance (a measure of how two quantities move together) between how likely a token is to be accepted and how easy the next step becomes.

The author gives a hand-built two-step example where WTV and the true objective have opposite gradients (+0.20 against −0.12) at the same point, so the surrogate would steer training away from better acceptance. A second toy example shows WTV ranking two draft models the wrong way round.

The proposed loss, REAL, estimates the true EAL on text sampled from the target. Each sampled token gets a weight that corrects for how the target and draft differ. To reduce noise, the final step of each term is replaced by its exact expected value, a standard statistical technique called Rao-Blackwellization. A small cross-entropy term (weight 0.1) is kept to stabilise training. Nothing else changes: not the drafter design, not the inference procedure.

The main test used a frozen Qwen3.6-35B-A3B target and a Domino drafter, trained for 3 epochs (about 140,000 steps) on prompts from the Open-PerfectBlend dataset with answers regenerated by the target. Evaluation ran in the vLLM serving engine on 9 task sets, including HumanEval, GSM8K, MATH-500 and BFCL, with 80 requests each and 5 seeds.

Under probabilistic serving at temperature 1, REAL averaged 4.792 accepted tokens per round. The author's reimplementations of four rival losses scored 4.657 (VSD), 4.715 (WTV), 4.740 (static LK) and 4.749 (DBLast). REAL or its variant without Rao-Blackwellization (4.791) was best on every task set. Under greedy serving, where the draft always proposes its top token, static LK was best at 4.544 against 4.537 for REAL. At temperature 0.5, static LK also edged ahead (5.002 against 4.993).

Rao-Blackwellization cut aggregate gradient variance by 52 to 58%. It made no difference to final acceptance length at temperature 1 but held up better at lower temperatures. Removing the cross-entropy term (4.624) or using cross-entropy alone (4.668) both did worse than REAL. Serving throughput was 4,662 tokens per second for REAL against 4,649 for DBLast and 4,553 for WTV. Training step times were similar across methods, at roughly 835 to 850 milliseconds.

## Our reading

In our reading, the strongest part of this paper is the diagnosis, not the benchmark. The argument that multiplying per-step acceptance rates is not the same as the real objective is simple, checkable and backed by worked examples. Anyone training draft models with a WTV-style loss has reason to look again at what that loss rewards. The author also shows that a WTV-shaped training metric favoured WTV while serving results favoured REAL, which is a useful warning about trusting training-time proxies.

The benchmark gains are real but modest. REAL beats the strongest rival by 0.043 accepted tokens per round, under 1%. The spread across seeds reaches 0.021, and no confidence intervals are reported. The consistency is encouraging: REAL led on every seed and every task set under sampled serving. But all baselines were reimplemented by the author at the loss level, with no described tuning, so the true gap to well-tuned rivals is unknown.

On readiness, we place this at level III, open source available. Results on public benchmarks meet level II, and a reference implementation of the loss is released as a branch of the Red Hat Speculators library. We have not checked the repository's licence or completeness, and no trained weights are mentioned. There is no documented production use.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/a-training-loss-that-targets-what-speculative-decoding-actually-rewards

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
