Together AI uses coding agents to keep re-tuning its inference engine
A preprint from Together AI describes a loop where AI agents rewrite and test the code that runs large models, again and again as traffic changes. The authors report up to 2.6 times higher throughput than the best public entries on a standard benchmark.

AI illustrationWhat the study shows
Together AI, a company that runs open AI models for customers, describes Together Compiler (TKCC). It is a loop in which AI coding agents rewrite the forward pass, the code that turns a request into the model's next output. Each rewrite is measured, and the fastest correct version is kept. The paper is a preprint and has not been peer reviewed.
The authors say the main obstacle was slow testing. Starting the serving engine for one large model took about 13 minutes before it could answer requests. Their fix is a persistent engine: the model is loaded once, and new code is swapped in without a restart. A fast tier runs a few model layers to catch crashes and slow code quickly. A slower tier runs all layers for accurate timings. The authors report this lets agents run about 252 times more experiments than a setup that restarts the engine for every test.
On SemiAnalysis' InferenceX benchmark, which replays real agentic coding traffic, TKCC beat the best listed configurations for four recent open models. These ran on NVIDIA, AMD and Amazon Trainium chips. The gain was up to 2.6 times in tokens per second. Each setting got 24 hours of optimisation on at most two server nodes.
A separate cost model estimates how best to split a model across chips, known as the parallelism configuration. For one model serving a single user, it picked a different split from the leading entry. With TKCC's code, that setup produced about 1.8 times the best published speed.
The authors also report that agents did better with less human help. Removing an expert-written knowledge base and expert strategy prompts made the agents faster in their tests. Giving agents plain engine source code worked better than custom compiler layers.
Outside language models, TKCC sped up eight science models, in weather, biology, time series and physics simulation, by 1.5 to 6.4 times. The outputs stayed bit-for-bit identical.
Our reading
In our reading, the most useful idea here is plain engineering, not clever AI. Agents are only as good as the feedback they get. Fast tests on random data mislead them, and slow tests on real data starve them. The paper shows concrete cases of both. An agent-written kernel that took 20 microseconds on random inputs took up to 62 milliseconds on real data. Keeping the engine running and swapping code in is a simple way to get feedback that is both fast and realistic.
The benchmark results are strong but need context. TKCC had a dedicated 24-hour budget per setting, double the usual hardware, and started from the leading entry's setup. The public baselines were not given the same treatment. The authors did check that they could reproduce the published baseline throughput, which helps. On our readiness scale this work sits at level II, Benchmarked: there are public benchmark results, but no released code.
The finding that expert-written guidance slowed the agents is interesting, but it rests on thin evidence. The main ablation covers one layer of one model on one GPU setup, over ten trials, with no repeated runs shown. It is a useful prompt for teams building agent tools. It is not yet a general rule.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | Working system with a described architecture, implementation snippet and ablations. |
| II Benchmarked | Met | Public benchmark results: SemiAnalysis InferenceX (agentic traces, pinned load generator), with baseline numbers reproduced to within about 2% on P90 interactivity. Also compared with open megakernel repositories on Llama 1B. Results are self-run. |
| III Open source | Not yet | No link to code, weights or data for TKCC in the paper. Only the baselines' repositories and container images are cited. |
| IV In production | Not yet | The authors state it serves trillions of tokens per day and compiles Together AI's fleet, and the Kimi K3 cost model gain is stated. But there is no named customer, no independent documentation and no production metrics, so this is a company claim and not yet strong evidence. |
What would move it up
Releasing the code (the hot-reload engine, cost model and agent harness) would reach level 3. Published production before-and-after numbers, or independent third-party confirmation of the fleet use and benchmark gains, would support level 4.
Checked automatically every week, last on 16 November.
The study
Reyna Abhyankar, Dylan Lim, Drew Wadsworth et al.. Together Compiler: Continual Compilation for AI Inference. arXiv, 10 November 2026. Read the paper
Notes
- Abhyankar, Lim et al. Together Compiler: Continual Compilation for AI Inference. arXiv preprint 2611.14681, November 2026. Not peer reviewed. Licence CC BY-NC-SA 4.0. Source
- No code for TKCC is linked in the paper. Readiness: level II, Benchmarked (public InferenceX results, self-run).
- SemiAnalysis InferenceX benchmark dashboard, used for the main comparison. Source
- Exact figures: engine cold start 761 s on MiniMax-M3.1-Flash with TensorRT-LLM; two-tier setup ran 1,354 candidates against 168 for one slow tier and found a 7.2% faster solution on B200 GLM 5.3; batch size 1 results of 1,111 TPS (MiniMax M3.1 Flash, B200, against 633) and 1,205 TPS (DeepSeek v4.1 Flash, GB300, against 943); Llama 1B up to 1.7x over megakernel baselines.
- Science models: speedups from 1.5x (Boltz-2) to 6.4x (Poseidon-T); Aurora temperature RMSE fell from 0.60 K to 0.55 K with 14 samples instead of 4 in fixed time.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

