Agent skills rewritten without feedback lose 15 points on SkillsBench
A preprint finds that when AI models repeatedly tidy up expert-written agent instructions with no testing, the instructions get worse. The first knowledge to go is the part the model could not have written itself.

AI illustrationWhat the study shows
Agent skills are short documents, usually written by an expert, that an AI agent loads when a task needs them. They hold procedures: which steps to take, which tools or scripts to call and which mistakes to avoid. Skills often need updating, and recent methods leave that job to a language model, frequently with no way to test the result.
The authors, from the Hong Kong University of Science and Technology, tested this "feedback-free" rewriting directly. Six models each rewrote expert skills 30 times in a row. Each round, the model was asked to find problems and improve the skill. It saw no task, no test result and no judge. The same model then used its rewritten skills as an agent.
The tests used three public benchmarks. SkillsBench has 87 multi-step tasks in software containers, each checked automatically. LogicBench and MedCalc-Bench pose logic questions and medical calculations, with the skill placed in the prompt. The authors report that every model did worse with its rewritten skills on all three. On SkillsBench, pass rates fell by about 15 percentage points on average after 30 rounds, and about half of that loss came in the first rewrite. On LogicBench and MedCalc-Bench, the rewritten skills ended up worth about as much as no skill at all.
To explain why, the authors asked each model to write ten skills of its own from a one-sentence topic, without seeing the expert's version. They call this the model's prior: what it already knows about the topic. Rewritten skills drifted towards these self-written versions. Item by item, expert content that none of the ten self-written skills contained was mostly lost: only about a quarter survived 30 rounds. Content that all ten contained mostly survived.
Putting the lost expert-only content back recovered about 60% of the SkillsBench loss. That was twice as much as restoring the other lost content. The authors then built SkillDiff, which tags the expert-only items in the original skill and adds a note asking the agent to follow them as written. Without changing a word of the skill, it raised SkillsBench pass rates by about 5 points on average, with gains for all six models. The paper is a preprint and has not been peer reviewed.
Our reading
In our reading, the core finding is simple and useful: an expert's skill document is valuable mainly for what the model does not already know. A model asked to improve the document has no source for that knowledge except the document itself. So repeated tidying pulls the text towards the model's own habits and sheds the unusual details first.
A concrete case makes the risk clear. Imagine a skill for preparing a financial report that says to call one internal script with an unusual flag, and to skip a step for one account type. A model asked to "improve" the skill may replace this with a cleaner, generic procedure. The document then looks better and works worse. The study's design, with automatic task checks, catches exactly this kind of silent regression.
On our readiness scale, we rate this work at level III, open source available. It reports results on public benchmarks, and the authors link a project repository. We have not checked what the repository contains or its licence. There is no documented use in production yet.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The work is a systematic lab study of controlled rewriting, with a proposed method (SkillDiff) and analysis. |
| II Benchmarked | Met | Results on public benchmarks: SkillsBench, LogicBench and MedCalc-Bench (the latter two from SRA-Bench), plus tau2-bench in the appendix. SkillDiff's +4.8 points is on SkillsBench against the original skill. |
| III Open source | Met | The abstract gives a project repository at https://github.com/LitaoGuo/How-Self-Evolution-Reshapes-Skills. The paper does not state its licence or exactly what it contains (code, data or weights), and I could not check it. |
| IV In production | Not yet | No documented use by a named organisation in production. All evidence is from benchmarks. |
What would move it up
Level 4 would need documented use of SkillDiff or the rewrite-protection idea in a named team's real agent workflow, with before and after results. Within research, it would help to test SkillDiff on more benchmarks, compare it with random-item tagging and other skill-improvement methods, and show a feedback-free rewriter that keeps items outside the prior.
The study
Litao Guo, Man Chen, Xinli Xu et al.. How Feedback-Free Self-Evolution Reshapes Agent Skills: A Systematic Study. arXiv, 19 November 2026. Read the paper
Notes
- Guo, Chen, Xu, Zhou, Zhang and Chen, "How Feedback-Free Self-Evolution Reshapes Agent Skills: A Systematic Study", arXiv preprint, November 2026, not peer reviewed. Source
- Project repository named in the paper. Contents and licence not checked by us. Source
- SkillsBench average loss after 30 rounds: 14.6 points; after round 1: 7.1 points; per-model range at round 30: 10.3 to 23.0 points (Table 4).
- LogicBench mean accuracy: 86.2% with the original skill, 71.4% after 30 rounds, matching the no-skill level. MedCalc-Bench: 89.7% to 78.7%, below 81.1% with no skill.
- Item retention after 30 rounds, averaged across models and benchmarks: 23.1% of items no self-written skill states, against 84.2% of items all ten state.
- SkillsBench restoration (Table 2): expert-only items recover 8.8 of 14.6 lost points; other lost items 4.4; all lost items 10.2.
- SkillDiff gains on SkillsBench (Table 3): +12.6, +5.7, +3.4, +2.3, +3.4 and +1.1 points, average +4.8. Cost: one topic call, ten generation calls and ten judge calls per skill (Appendix C.4).
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections
