Claude Code agent designs LLM quantisation recipes that beat public baselines

An MIT and NVIDIA preprint lets a coding agent search for the best way to compress each reasoning model. It gains about one to two accuracy points over the strongest public recipes for an 8-hour search on 8 GPUs, and slightly beats NVIDIA's own compressed release of a 320B model.

6 min readIIBenchmarkedPreprint, not peer reviewed
Save
Figure 1 from the paper "QDA: Closed-Loop LLM Quantization Design with Agents"
Average accuracy of four reasoning models at three compression levels, comparing the strongest public recipe with the agent-designed recipe, alongside the GPU time and agent cost of one search. Figure 1 from Xingyang Li et al., 2026, arXiv:2611.23388. CC BY 4.0.

What the study shows

Quantisation stores a model's numbers in fewer bits, for example 4 instead of 16, so it needs less memory and runs faster. The catch is lost accuracy. Many published methods reduce that loss, and they can be stacked. The exact mix of methods, settings and calibration data is called a recipe.

The authors, from MIT and NVIDIA, first show that no single recipe wins everywhere. On one Qwen model, the AWQ method beat GPTQ by about 5 points at 4-bit but lost by about 12 points at 3-bit. So each new model and precision needs its own search.

Their system, called QDA, hands that search to Claude Code running Claude Opus 5.5. The agent proposes a recipe, a shared library builds it, and an evaluator scores it. It then decides what to change next. Each search had 64 GPU-hours and cost $19 to $75 in agent fees.

The key trick is a fast score. Fully testing a compressed reasoning model takes 23 to 86 GPU-hours. Instead, the evaluator measures how far the compressed model's next-word predictions drift from the original's, a quantity called KL divergence. This takes minutes. The texts it uses are hidden from the agent, so the agent cannot game them. For a few promising candidates, the agent also asks for long answers to catch failures such as endless loops.

The authors report that across four reasoning models, QDA beat the strongest public recipe on average by 0.9 points at 4-bit weights, 1.4 points when activations are also cut to 8-bit, and 2.2 points at 3-bit weights. Tests covered maths, science, coding and instruction following.

On the 320B-parameter GLM-5.3-Flash, compressed to NVIDIA's 4-bit NVFP4 format, one 6.5-hour search kept 98.3% of the original's accuracy. NVIDIA's own release kept 96.9% and Red Hat AI's 96.3%. All QDA models run on existing serving kernels with no added slowdown.

This is a preprint and has not been peer reviewed. On our readiness scale it sits at level II, Benchmarked: the results use public benchmarks and public baselines, but the code is not yet available.

Our reading

In our reading, the headline gains are modest but real where it counts. At 4-bit, one point of accuracy is close to the noise of a single search. At 3-bit, where compression usually hurts most, the gains are larger and more convincing. The clearest case is the DeepSeek-R1-Distill-Llama-8B model, where QDA beat plain GPTQ by about 6 points at 3-bit.

The more useful lesson may be about the setup rather than the agent. When the authors gave the same agent NVIDIA's ModelOpt toolkit instead of their library and evaluator, it did worse, even though it was allowed to select on the test benchmarks. When they swapped the agent for a simple automated search over the same library, it lost only up to about one point. The fast hidden score and the reusable library do much of the work.

A practical example: a team that wants to serve a newly released open model at 4-bit could run one overnight search on a single 8-GPU server. It would get a checkpoint that slots into its existing vLLM setup, instead of waiting weeks for a hand-tuned recipe.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

II
Benchmarked
LevelStatusEvidence
I  Lab onlyMetThe paper describes a working system and runs it on four models and a 320B-parameter model.
II  BenchmarkedMetResults are on public benchmarks (AIME 2025, HMMT Feb 2026, GPQA-Diamond, LiveCodeBench v6, IFBench, SuperGPQA) against public baselines (GPTQ, AWQ, ParoQuant, GSQ, QuaRot+GPTQ) and vendor checkpoints from NVIDIA and Red Hat AI.
III  Open sourceNot yetThe abstract says 'We will release the library, the evaluator and the harness', and Appendix E says the full task statement 'ships with the artifact'. No link or release is given in the paper, so nothing is verifiable yet.
IV  In productionNot yetNo named organisation is reported using QDA in production. The NVIDIA and Red Hat checkpoints are baselines, not QDA users.

What would move it up
A public release of the library, evaluator, harness and task statements would reach level 3. Level 4 needs documented use by a named organisation to ship quantized checkpoints, ideally with independent replication, repeated searches on more models, and different agents.

The study

Xingyang Li, Tianlang Zhao, Song Han et al.. QDA: Closed-Loop LLM Quantization Design with Agents. arXiv, 17 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY 4.0

Notes

  1. Li, Zhao, Han and Zhu, 'QDA: Closed-Loop LLM Quantization Design with Agents', arXiv preprint, 17 November 2026, CC BY 4.0. Source
  2. Main results (Table 2): average gains over the strongest public recipe of 0.9, 1.4 and 2.2 points at W4A16, W4A8 and W3A16 across Qwen3.5-9B, Qwen3.6-35B-A3B, Qwen3.8-27B and DeepSeek-R1-Distill-Llama-8B.
  3. GLM-5.3-Flash at NVFP4 (Table 3): 98.3% of FP8 accuracy for QDA, 96.9% for NVIDIA's release, 96.3% for Red Hat AI's; checkpoint sizes 183.6, 190.4 and 184.3 GiB.
  4. KL score costs minutes and is 350 to 990 times cheaper than a full evaluation of 23 to 86 GPU-hours; it ordered 82% of 44 checkpoint pairs like accuracy, against 73% for layer-wise error.
  5. Code: the authors say they will release the library, evaluator and harness. No link was available at the time of writing.
  6. Readiness: level II, Benchmarked. Public benchmark results against public baselines; no released code or documented production use.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Efficiency