# Meta's refusal-driven LLM dispatcher served a classifier on 43% fewer H100s

A Meta preprint describes a production traffic shaper that sends AI requests only as fast as backends accept them. In a same-traffic test it used far fewer GPUs, but low-priority work waited much longer at peaks.

- Category: In industry
- Readiness level: IV (In production) on Science Underway's four-step scale, as proposed
- Published: 2026-11-19
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/meta-s-refusal-driven-llm-dispatcher-served-a-classifier-on-43-fewer-h100s

## The study

Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity (preprint, not peer reviewed). arXiv 2611.24891. https://arxiv.org/abs/2611.24891

## At a glance

- What it shows: Meta runs a system that holds waiting AI requests in a queue and sends them only as fast as the computers behind it say they can accept. In a side-by-side test on live traffic, it did the same classification job on 39 high-end GPUs instead of 68.
- What it does not show: It does not show the method works for chat or AI agents that people wait on, or outside Meta's own setup. The saving also had a cost: the least urgent work waited hours longer at busy peaks.
- Why it matters: GPUs are scarce and slow to add, so keeping each one fully used matters. This is a documented case of a simple control rule doing that at very large scale, without constant hand-tuning.

## What the study shows

The paper describes how Meta dispatches large language model (LLM) requests that do not need an instant answer, such as understanding posts and live video. Its traffic shaper carries more than 300 million such requests a day. In its busiest 24 hours, during a bulk job over existing content, it carried about 1.1 billion.

The design has two parts. CAP is a quick check at each backend that refuses extra work with a standard 'too many requests' error (HTTP 429) instead of letting it wait inside the server. TRACK is a controller that raises the sending rate by 3% every 15 seconds and cuts it when refusals rise. Work that cannot be sent yet waits in a durable priority queue, one that survives crashes and serves urgent items first.

The strongest evidence is two concurrent production A/B tests on the same traffic. For a classifier model, the shaper used 39 Nvidia H100 GPUs against 68 for the older hand-tuned system, with matching coverage and scores. The authors attribute 5 of the 29 saved cards to the old system's safety margin. On equal fleets of about 600 AMD MI300X GPUs, the shaper cleared every shared backlog first, by a median of 3 hours. Its engines ran nearly full batches, a median 124 of 128 slots against 41, and served 9% more requests per GPU.

The saving was not free. On the two peak days of a four-day bulk job, the lowest-priority classifier work waited a median of about 6.6 hours, against about half an hour before. The two lowest priority bands met their freshness goals for only about half of items in that period.

In controlled replay on real inference engines with steady capacity, TRACK delivered about the same throughput as a fixed rate set 20% above capacity, but drew about a quarter as many refusals. After a simulated loss and return of capacity, TRACK took about 5 minutes to climb back. Without a per-server cap, replay at twice capacity saw goodput (requests finished within their timeout) fall to zero and stay there.

This is a preprint and has not been peer reviewed. The authors are Meta employees reporting on their own system.

## Our reading

In our reading, the most useful idea here is a division of labour. The backend decides how much it can take, and says so with a fast refusal. The dispatcher never tries to predict request size, which is hard for LLMs because output length is unknown in advance. It simply follows the refusals. This is the same logic networks use to avoid congestion, applied to GPU fleets.

The headline 43% figure should be read with care. The old system had no cap at its engines and began to fail at about two thirds of its profiled rate. So part of the gain likely comes from adding caps, not from TRACK's control rule. The paper does not split the saving between the two. The authors also say plainly that a well-tuned fixed rate can match or beat TRACK while capacity holds steady. TRACK's value is that it keeps up when capacity moves, which the authors report it did through nine fleet resizes without retuning.

A concrete example: a team running nightly tagging of uploaded images on a shared GPU pool could size that pool for average load, not peak. Urgent items would still be handled quickly. Bulk catch-up work would wait in the queue and drain when capacity frees up. The trade-off is that the bulk work arrives later, sometimes hours later.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/meta-s-refusal-driven-llm-dispatcher-served-a-classifier-on-43-fewer-h100s

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
