# Token spend tracks how much engineers use AI agents, not what they ship

A five-week study of 404 engineers at JetBrains finds that an engineer's AI token bill says little about how much code they commit. The paper is a preprint and has not been peer reviewed.

- Category: In industry
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-15
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/token-spend-tracks-how-much-engineers-use-ai-agents-not-what-they-ship

## The study

The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.13396, arXiv 2611.13396. https://arxiv.org/abs/2611.13396

## At a glance

- What it shows: At one software company, the amount engineers spent on AI coding agents barely changed with how much code they delivered. The same people were heavy users week after week, and many used agents for research, analysis and planning as well as coding.
- What it does not show: It does not show that spending more on AI causes more or less output, or whether the extra use was worthwhile. It covers one company over five weeks.
- Why it matters: Some companies rank engineers by AI spend or set minimum spending targets. This study suggests the number mostly reflects personal habit and type of work, not delivery.

## What the study shows

Coding agents are AI tools that read code, run commands and make changes for an engineer. Their providers bill by the token, a small unit of text the model reads or writes. An engineer's token spend is the total of those units over a period.

The authors are from Carnegie Mellon University, JetBrains and the University of California, Irvine. They linked five weeks of token counts from JetBrains' internal AI gateway to the commits of 404 engineers. A commit is a saved change to the code. They also analysed an anonymous company survey of 412 employees about AI use.

The main comparison holds each engineer fixed and compares their weeks with one another. The authors report that a week with twice an engineer's usual spend had only about 10% more commits. If spend tracked output closely, doubling one would roughly double the other.

The authors tested several explanations for the weak link. They tried lines changed instead of commits, days instead of weeks, and allowed for delays between spend and commits. None of these closed the gap.

Spend varied far more between people than with their output. Among weeks with similar commit counts, the heaviest-spending tenth used 39 to 240 times the tokens of the lightest tenth. Commit-based measures explained under two fifths of the differences between engineers. Role and team added nothing useful.

The authors estimate that between about 4% and about two fifths of spend does not vary with the week's commits. The upper figure also includes the fixed cost of using an agent at all and spend in weeks when the code was written by hand.

In the survey, technical staff said they had used the coding agents for something other than coding in nearly nine of ten evaluations. Common uses were data analysis, brainstorming, research and planning. Most reported saving time, and those with broader use reported saving more hours. These are self-reports.

The authors recommend using token spend only for resource accounting. They advise against using it to judge engineers' output or efficiency.

## Our reading

In our reading, the comparison with lines of code holds up, with one twist. Lines of code at least count something produced. Tokens count an input, recorded whether or not anything useful comes out. A token spent on a prototype that shaped a design looks the same on the bill as one spent on output that was thrown away.

The study's strongest feature is its design. Earlier figures came mostly from analytics vendors comparing groups of heavy and light users. Those comparisons mix up the person with the spend. Here each engineer is compared with themselves, and the link is still weak. So the problem is not just that different engineers work differently.

A concrete example shows the risk. Suppose a manager divides an engineer's token bill by their commits to judge efficiency. In a busy week, the engineer will look about 40% more efficient without changing how they work. When a new model arrives, the same ratio can rise with no change in output, because newer models used more tokens for the same commits. The headline caveat is that these are associations at one company that builds AI coding tools.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/token-spend-tracks-how-much-engineers-use-ai-agents-not-what-they-ship

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
