Cyclomatic complexity feedback makes coding agents write simpler, not more correct, code
A preprint tests short code-quality reports injected into coding agents' context during long runs on Commit0, SlopCodeBench and SWE-EVO. The agents produced structurally simpler code, but held-out tests did not show more correct code.


What the study shows
Coding agents now work for hours on their own, and they mostly judge success by whether tests pass. Passing tests say little about structure. An agent can pile logic into a few huge functions and then keep building on them. The authors, from Nera Software, Cohere Labs and Mila, test a simple fix. After every turn in which the agent changes code, a tool measures the changed functions and adds a short report to the agent's context before it continues.
The report uses cyclomatic complexity, a decades-old measure that counts the independent paths through a function. More paths means code that is harder to follow and test. The report names every changed function above a limit of 6 and asks the agent to bring it down. A second report lists changed lines that the agent's own tests never run. The tool needs no training, no reference solution and no access to the hidden tests.
The design is careful about one trap. A message after every edit might change behaviour whatever it says. So the main comparison is against a reminder arm that gets the same refactoring instructions on the same schedule, but no measurements. A fifth arm gets no message at all. Four models were tested: DeepSeek V4.1 Flash, GLM-5.3 Flash, Claude Haiku 5.5 and GPT-6 Luna. The three benchmarks ask agents to rebuild Python libraries (Commit0), extend a program over several rounds of changing specifications (SlopCodeBench), or update a real project to its next release (SWE-EVO).
The main outcome is erosion: the share of a codebase's complexity that sits in functions with complexity above 10. The authors report that complexity feedback cut erosion for all four models on Commit0 and SlopCodeBench. On SlopCodeBench it removed nearly all of it, holding erosion at about 2% or less at every checkpoint. On SWE-EVO, where agents edit mostly existing code, the drop was small and clear for only two of four models.
Agents did act on the reports. Under complexity feedback they reworked most flagged functions, and on SlopCodeBench well over 90% ended under the limit. With the reminder alone, only 1 to 18% of over-limit functions ended under it.
Correctness, measured on tests the agent never saw, showed no detectable change for any model. Pooled estimates leaned slightly positive on Commit0 and SWE-EVO and slightly negative on SlopCodeBench. All their ranges included zero. One extra analysis on SlopCodeBench suggests a small cost cannot be ruled out. The paper is a preprint and has not been peer reviewed.
Our reading
In our reading, the strongest result is behavioural. Agents read the reports and followed them, and the gap over the reminder arm is large and consistent on two benchmarks. A mechanical number, delivered at the right moment, steered models from four different developers. That is useful evidence that in-run feedback works as a control lever, not just end-of-run grading.
The weaker part is what the lower number means. The agent is steered towards complexity, and erosion is measured on complexity. Splitting a big function into helpers lowers the score without removing any branching logic. The authors check this. When they fold single-use helpers back into their callers, much of the gain disappears: between 3 and 46% of it survives on Commit0. On SlopCodeBench the total amount of branching did fall, by about 14 to 21% for three models, which points to some real simplification there.
A practical example: a team that lets an agent build an internal library overnight could add a complexity check after each edit at little cost. The feedback text was about 1% of tokens or less. The study suggests the morning's code would have fewer sprawling functions. It does not yet show that the code would have fewer bugs. On our readiness scale this is level II, benchmarked: public benchmark results, but no released code for the measuring tool and no documented production use.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | Working method with a defined intervention, five arms and a full experimental design. |
| II Benchmarked | Met | Results on three public benchmarks (Commit0, SlopCodeBench, SWE-EVO) with four models and paired 95% intervals, against a reminder-only control and a no-message baseline. |
| III Open source | Not yet | The paper describes the SlapKit measurement tool but gives no link to its code, prompts or run data. The links given are to the benchmarks, the Oh-My-Pi harness and the lizard library, which are third-party. |
| IV In production | Not yet | No documented use by a named organisation in production or on real codebases. |
What would move it up
Releasing SlapKit and the feedback messages as open source would reach level 3. Level 4 would need a named team using it on real, long-lived codebases, ideally showing lower maintenance cost or fewer defects. A new benchmark of long change sequences, and an arm that states the limit without measuring code, would test the gaps the authors name.
The study
Nicolas Chapados, Jean-François Gagné, Nouha Dziri. Metrics-in-the-Loop: Complexity Feedback Reshapes Agent-Written Code. arXiv, 18 November 2026. Read the paper
Notes
- Chapados, Gagné and Dziri, Metrics-in-the-Loop: Complexity Feedback Reshapes Agent-Written Code, arXiv preprint, November 2026, CC BY 4.0. Source
- Exact figures: pooled CT minus R erosion change was -0.078 on Commit0, -0.236 on SlopCodeBench and -3.85 thousandths on SWE-EVO. Pooled CT minus R correctness was +3.1 points [-0.1, +6.3] on Commit0, -1.7 [-3.6, +0.1] on SlopCodeBench and +2.7 [-0.2, +5.6] on SWE-EVO fix rate. Mixed-model odds ratio on SlopCodeBench: 0.82 [0.70, 0.96].
- Commit0 benchmark. Source
- SlopCodeBench benchmark. Source
- SWE-EVO benchmark. Source
- Oh-My-Pi agent harness used in the study (third party). Source
- lizard, the open-source library used to compute cyclomatic complexity. Source
- No code release for the SlapKit measurement tool is given in the paper. Readiness: level II, benchmarked.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections


