# Together AI uses coding agents to keep re-tuning its inference engine

A preprint from Together AI describes a loop where AI agents rewrite and test the code that runs large models, again and again as traffic changes. The authors report up to 2.6 times higher throughput than the best public entries on a standard benchmark.

- Category: Efficiency
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-14
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/together-ai-uses-coding-agents-to-keep-re-tuning-its-inference-engine

## The study

Together Compiler: Continual Compilation for AI Inference (preprint, not peer reviewed). arXiv 2611.14681. https://arxiv.org/abs/2611.14681

## At a glance

- What it shows: Together AI built a system where AI coding agents keep rewriting the software that runs large language models, test each change on a live setup and keep the fastest version. On a public benchmark it beat the best published results by up to 2.6 times.
- What it does not show: Nobody outside the company has checked these results, and the code is not released. The paper does not report long-running tests of the system adapting as real traffic shifts.
- Why it matters: Running AI models is a large and growing cost. If agents can tune serving software in a day for each new model, providers could serve more users on the same hardware.

## What the study shows

Together AI, a company that runs open AI models for customers, describes Together Compiler (TKCC). It is a loop in which AI coding agents rewrite the forward pass, the code that turns a request into the model's next output. Each rewrite is measured, and the fastest correct version is kept. The paper is a preprint and has not been peer reviewed.

The authors say the main obstacle was slow testing. Starting the serving engine for one large model took about 13 minutes before it could answer requests. Their fix is a persistent engine: the model is loaded once, and new code is swapped in without a restart. A fast tier runs a few model layers to catch crashes and slow code quickly. A slower tier runs all layers for accurate timings. The authors report this lets agents run about 252 times more experiments than a setup that restarts the engine for every test.

On SemiAnalysis' InferenceX benchmark, which replays real agentic coding traffic, TKCC beat the best listed configurations for four recent open models. These ran on NVIDIA, AMD and Amazon Trainium chips. The gain was up to 2.6 times in tokens per second. Each setting got 24 hours of optimisation on at most two server nodes.

A separate cost model estimates how best to split a model across chips, known as the parallelism configuration. For one model serving a single user, it picked a different split from the leading entry. With TKCC's code, that setup produced about 1.8 times the best published speed.

The authors also report that agents did better with less human help. Removing an expert-written knowledge base and expert strategy prompts made the agents faster in their tests. Giving agents plain engine source code worked better than custom compiler layers.

Outside language models, TKCC sped up eight science models, in weather, biology, time series and physics simulation, by 1.5 to 6.4 times. The outputs stayed bit-for-bit identical.

## Our reading

In our reading, the most useful idea here is plain engineering, not clever AI. Agents are only as good as the feedback they get. Fast tests on random data mislead them, and slow tests on real data starve them. The paper shows concrete cases of both. An agent-written kernel that took 20 microseconds on random inputs took up to 62 milliseconds on real data. Keeping the engine running and swapping code in is a simple way to get feedback that is both fast and realistic.

The benchmark results are strong but need context. TKCC had a dedicated 24-hour budget per setting, double the usual hardware, and started from the leading entry's setup. The public baselines were not given the same treatment. The authors did check that they could reproduce the published baseline throughput, which helps. On our readiness scale this work sits at level II, Benchmarked: there are public benchmark results, but no released code.

The finding that expert-written guidance slowed the agents is interesting, but it rests on thin evidence. The main ablation covers one layer of one model on one GPU setup, over ten trials, with no repeated runs shown. It is a useful prompt for teams building agent tools. It is not yet a general rule.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/together-ai-uses-coding-agents-to-keep-re-tuning-its-inference-engine

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
