AI agents find bugs alone; CVE records double, fix times hold
A preprint from NIST and university researchers maps what AI agents have shown they can do in software security since 2024. It then checks a decade of public vulnerability records for the effects.


What the study shows
The authors built a rating scale for AI work on software flaws, modelled on the levels used for self-driving cars. It runs from A0, where the system does nothing, to A5, where an agent picks its own targets and works for weeks. Each result also gets a precision grade, meaning how often its findings hold up. It also gets an access tier, from lab-only to open weights (models anyone can download), and an evidence grade.
They rated four tasks: finding a flaw, proving it is real, writing a fix, and checking that a fix is correct. AI agents gathered and scored over 53,000 sources, and two authors reviewed every result. The final map has 128 dated steps, of which 85 rest on direct evidence from real software.
The authors report that agents did all four tasks with no human help by October 2025. That was before restricted frontier models such as Anthropic's Claude Mythos Preview arrived in 2026. Finding and proving flaws has since reached expert-level precision at every access tier. For example, a security firm confirmed 96% of findings from Aikido Attack, a commercial agent anyone can sign up for.
Wider access followed fast. Open-weight models reached the same levels about 10 to 14 months after the first system, on real-software evidence. Across twelve security benchmarks, the gap was about 8 months.
The authors then turned to public records. CVE records, the standard public list of disclosed software flaws, more than doubled in the first ten months of 2026 against the same months of 2025. The share rated Critical or High rose from 46% to 56%. Where the age of the flawed code is known, severe bugs at least ten years old were reported 6 to 11 times as often.
Some things did not change. The median time from report to fix at the Zero Day Initiative, a large disclosure programme, stayed between 96 and 119 days every year since 2016. The median time from a CVE record to a confirmed attack stayed at 2 to 5 days.
The paper is a preprint and has not been peer reviewed. On our readiness scale it sits at level II, benchmarked: it is checked against public records and benchmarks, but we could not find its released data or code.
Our reading
In our reading, the most useful contribution is the separation of three questions that public debate tends to blur. What has an AI system been shown to do? Who can get hold of it? How good is the evidence? A vendor blog post and an independently confirmed result now sit on the same scale, with the difference made visible.
The headline result is a mismatch. Capability and volume have jumped, but the timelines that matter to defenders have not moved yet. The authors are careful here. They say the records cannot tell whether defenders absorbed the extra load or whether attackers simply have not adopted these tools at scale.
The fix-rate data is the part we would point a product lead to first. Only 5 to 26% of reports sent in bulk to outside maintainers were fixed, against 58 to 94% of curated reports. A flood of valid findings does little good if the people who own the code cannot act on them.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The work goes beyond lab-only. It is a systematisation and measurement study built on a large evidence base. |
| II Benchmarked | Met | Results are compared with public data. These include CVE records, Zero Day Initiative timelines and twelve public security benchmarks, with agreement statistics for the scoring (kappa 0.83 between the two authors). |
| III Open source | Not yet | The paper says the full table of readings is 'released with our artifact', but gives no link. We could not confirm that code, data or the pipeline are public. |
| IV In production | Not yet | This is a survey and measurement framework, not a deployed tool. The underlying systems it rates are used in production by named organisations, but the framework itself is not. |
What would move it up
Publishing the artifact with a working link would move it to level 3. That means the scored readings, the evidence pipeline and the record-analysis code. Reuse by a lab, regulator or evaluator to track capability would support level 4. Independent replication of the map would also help.
The study
Amine Lbath, Manan Suri, Aurelien Delaitre et al.. SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research. arXiv, 19 November 2026. Read the paper
Notes
- Lbath, Suri, Delaitre, Okun, Amini and Manocha, 'SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research', arXiv preprint, November 2026, not peer reviewed. Licence CC BY 4.0. Source
- The paper says the full table of readings is released with its artifact, but gives no link. We could not locate public code or data.
- CVE records counted from the public CVE List as of 31 October 2026: 39,638 records in January to October 2025 against 89,749 in the same months of 2026. Share rated Critical or High (CVSS 7 or more): 46% in 2025, 56% from July to October 2026. Source
- Open-weight delay on twelve benchmarks: 7.6 months, 95% bootstrap interval 5.4 to 10.0, fitted over 104 best scores.
- Fix rates by route (Table 6): bulk reports to outside maintainers 5 to 26% fixed; curated programmes and OSS-Fuzz 58 to 94%.
Edited and signed by
Stefan PerdalEditor and publisher89 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

