Meta's refusal-driven LLM dispatcher served a classifier on 43% fewer H100s

A Meta preprint describes a production traffic shaper that sends AI requests only as fast as backends accept them. In a same-traffic test it used far fewer GPUs, but low-priority work waited much longer at peaks.

6 min readIVIn productionPreprint, not peer reviewed
Save
A still-life photo of a wooden tray of seeds with a blue card gate, trickling into small glass jars.AI illustration
Illustration: AI generated

What the study shows

The paper describes how Meta dispatches large language model (LLM) requests that do not need an instant answer, such as understanding posts and live video. Its traffic shaper carries more than 300 million such requests a day. In its busiest 24 hours, during a bulk job over existing content, it carried about 1.1 billion.

The design has two parts. CAP is a quick check at each backend that refuses extra work with a standard 'too many requests' error (HTTP 429) instead of letting it wait inside the server. TRACK is a controller that raises the sending rate by 3% every 15 seconds and cuts it when refusals rise. Work that cannot be sent yet waits in a durable priority queue, one that survives crashes and serves urgent items first.

The strongest evidence is two concurrent production A/B tests on the same traffic. For a classifier model, the shaper used 39 Nvidia H100 GPUs against 68 for the older hand-tuned system, with matching coverage and scores. The authors attribute 5 of the 29 saved cards to the old system's safety margin. On equal fleets of about 600 AMD MI300X GPUs, the shaper cleared every shared backlog first, by a median of 3 hours. Its engines ran nearly full batches, a median 124 of 128 slots against 41, and served 9% more requests per GPU.

The saving was not free. On the two peak days of a four-day bulk job, the lowest-priority classifier work waited a median of about 6.6 hours, against about half an hour before. The two lowest priority bands met their freshness goals for only about half of items in that period.

In controlled replay on real inference engines with steady capacity, TRACK delivered about the same throughput as a fixed rate set 20% above capacity, but drew about a quarter as many refusals. After a simulated loss and return of capacity, TRACK took about 5 minutes to climb back. Without a per-server cap, replay at twice capacity saw goodput (requests finished within their timeout) fall to zero and stay there.

This is a preprint and has not been peer reviewed. The authors are Meta employees reporting on their own system.

Our reading

In our reading, the most useful idea here is a division of labour. The backend decides how much it can take, and says so with a fast refusal. The dispatcher never tries to predict request size, which is hard for LLMs because output length is unknown in advance. It simply follows the refusals. This is the same logic networks use to avoid congestion, applied to GPU fleets.

The headline 43% figure should be read with care. The old system had no cap at its engines and began to fail at about two thirds of its profiled rate. So part of the gain likely comes from adding caps, not from TRACK's control rule. The paper does not split the saving between the two. The authors also say plainly that a well-tuned fixed rate can match or beat TRACK while capacity holds steady. TRACK's value is that it keeps up when capacity moves, which the authors report it did through nine fleet resizes without retuning.

A concrete example: a team running nightly tagging of uploaded images on a shared GPU pool could size that pool for average load, not peak. Urgent items would still be handled quickly. Bulk catch-up work would wait in the queue and drain when capacity frees up. The trade-off is that the bulk work arrives later, sometimes hours later.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

IV
In production
LevelStatusEvidence
I  Lab onlyMetThe work goes well beyond a lab idea. It has a described design, replay experiments on real engines and an emulator.
II  BenchmarkedMetControlled replay on real vLLM engines compares TRACK with fixed rates and other controllers (Table C1), plus two same-traffic production A/Bs. The tests use Meta's own traffic and workloads, not a public benchmark, so this level is met in a loose sense.
III  Open sourceNot yetThe paper gives no link to code, weights or data. The traffic shaper and FOQS are internal to Meta. Only the general design is described.
IV  In productionMetDocumented use by a named organisation: Meta's traffic shaper carries over 300 million online requests a day for post and live-video understanding. It has run for months, with reported A/Bs, fleet resizes and a 15-week reservation record.

What would move it up
It is already at the top level on the scale. The scale treats the levels as separate evidence, though, and level III is not met. Released code or a reference implementation, for example an open dispatcher that works with vLLM's new queue limits, would fill that gap. Independent results from outside Meta, or tests on interactive traffic, would strengthen the claims.

The study

Benjamin Novotny, Doug Szeto, Chang Wang et al.. Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity. arXiv, 18 November 2026. Read the paper

Preprint, not peer reviewed · Licence arXiv (no reuse)

Notes

  1. Novotny et al., Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity. arXiv preprint, not peer reviewed. Source
  2. Classifier A/B: 39 H100s for the shaper against 68 for the precursor; 5 of the 29 saved cards are the precursor's margin. Lowest band waited a median of about 6.6 hours against about 0.5 hours on backfill peak days; the two lowest bands met goals for 44.2% and 55.1% of items over the backfill's last three days.
  3. Zero-shot A/B: about 600 MI300X per path over 23 days; backlogs drained a median 3 hours sooner; median 123.8 against 40.7 of 128 running sequences; 1.09 times the requests per GPU.
  4. Replay at stable ceiling (cap 40): TRACK 108.5 req/s per host, fixed 100% 107.0, fixed 120% 109.5; refusals on 3.8%, 1.1% and 15.5% of attempts. Recovery to 95% of full capacity about 220 s.
  5. Reserved capacity: 94.8% utilisation over first four full days against 88.2% for the static system (observational); 95.8% over 15 weeks after the provider's admission update; 99.1% in a backlogged week.
  6. vLLM v0.29.0 release notes, which add optional queue limits. Source

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on In industry