Study of 120,000 GitHub bug reports: AI adds detail, not more confirmed bugs
A preprint tracing AI use in bug reports on 101 popular open-source projects, including the Linux kernel, finds AI-involved reports are fuller but no more often confirmed. AI help with reproducing and diagnosing bugs is the use linked to valid findings.

AI illustrationWhat the study shows
The authors, from the University of Hong Kong and other Chinese institutions, studied 120,918 bug reports on 101 open-source projects. The projects are popular GitHub repositories plus the Linux kernel. Reports were opened between January 2024 and September 2026. The paper is a preprint and has not been peer reviewed.
They sorted reports by AI's role. In 'AI-assisted' reports, a person hit the bug and used AI to help. In 'AI-discovered' reports, an AI tool found the bug, usually by reading source code. A third group was only suspected of AI use. Most labels were applied by a large language model (LLM), an AI system trained on text, and disagreements with a second LLM were settled by human reviewers.
AI use rose quickly. About 1 in 140 bug reports involved AI in 2024. By mid 2026 it was about 1 in 12. Reports where AI found the bug grew from a small fraction to about a fifth of AI-involved reports.
AI-involved reports were more complete. They more often included reproduction steps, triggering conditions and the expected behaviour. Comparing reports within the same project and quarter, about 9 percentage points more were rated high quality.
Completeness did not mean correctness. Overall, maintainers confirmed AI-involved reports as real bugs at about the same rate as other reports. Reports where AI found the bug did worse, and most of them never got a clear verdict at all.
The type of AI help mattered. The authors report that AI help with reproducing a bug roughly doubled the odds of a valid verdict. AI help with diagnosis also helped. AI help with writing the report showed no link to validity.
AI-generated materials still needed checking. Maintainers rejected about a quarter of explicitly AI-generated reproducers, the small scripts or inputs that trigger a bug, against about 1 in 20 others. That figure rests on only 34 AI-generated reproducers.
Maintainers are also starting to use AI themselves, mostly to read code and suggest fixes. AI-involved maintainer replies rose to about 3% of replies in 2026, from almost none in 2024. Most came from bots, and a handful of projects accounted for most of them.
Our reading
In our reading, the central result is a gap between polish and truth. AI makes reports look complete. It does not, on its own, make them right. The uses of AI that track with confirmed bugs are the ones that test something: building a reproducer, or digging into the cause. Writing nicer prose is not one of them.
A practical example: a developer whose program crashes asks an AI assistant to write up the bug. The result reads well, but the evidence suggests it is no more likely to be confirmed. If the developer instead has the AI build a small script that triggers the crash, and runs it, the report is more likely to land.
The split between AI-assisted and AI-discovered reports is the second lesson. When a person hit the bug, AI help drew more maintainer work, not less. When an AI found the bug by reading code, maintainers mostly did not engage. Fewer than one in five such reports got any reply. That is a lack of attention, not proof the findings were wrong.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | This is an empirical measurement study rather than a new method. The measurement framework and annotation approach are described in detail, with codebook definitions in the appendix. |
| II Benchmarked | Met | The paper reports quantitative results on a large real-world dataset of 120,918 reports from 101 projects. This is a measurement study rather than a benchmark of a tool, and the labels come from LLM annotation with limited accuracy reporting, so this level is met only loosely. |
| III Open source | Met | The authors state that code and data are available at https://github.com/Norrrrrrr-lyn/AI-Generated_Report. We could not check the repository's contents or licence. |
| IV In production | Not yet | The recommendations have not been shown to work in production. The only application is the authors' own submission of 53 fuzzing reports, with no comparison group and no named adopting organisation. |
What would move it up
Independent replication on other projects, with accuracy of the LLM labels reported against human-labelled samples. A controlled test of the reporting checklist, with matched reports submitted with and without it. Named maintainers or projects adopting the guidance and reporting results would support the top level.
The study
Yinuo Liu, Yunlong Lyu, Zichao Luo et al.. Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance. arXiv, 19 November 2026. Read the paper
Notes
- Liu, Lyu, Luo, Dong, Luo, Chen and Chen, 'Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance', arXiv preprint 2611.25605, 19 November 2026. Not peer reviewed. Source
- Code and data, as stated by the authors. Not checked by us. Source
- Exact figures: 3,000 AI-involved reports of 120,918 (2.5%): 2,508 AI-assisted, 396 AI-discovered, 96 suspected. Share of all bug reports 0.7% (2024), 1.9% (2025), 8.3% (2026-Q3), 11.1% (September 2026). AI-discovered share of AI-involved reports 3.0% (2025) to 19.2% (2026).
- Completeness: 50.70% high quality for AI-involved against 33.77% for other reports; adjusted gap +8.72 points. Validity among decided reports: adjusted gap -0.37 points, not significant. AI-discovered valid rate -10.92 points (q = 0.033), 62.37% uncertain.
- Regression on 68,773 decided reports: odds ratios 2.25 (reproduction and testing), 1.42 (problem diagnosis), 1.05 (report writing, not significant). Reproducers rejected: 23.53% AI-generated (n = 34) against 5.14% other.
- Maintainer actions per report: +0.46 for AI-assisted, -0.93 for AI-discovered. AI-involved maintainer replies 0.07% (2024) to 3.40% (2026); bots 65.4%, humans using AI 34.6%; five projects 63.6% of 2026 total. Authors' test: 53 reports, 69.81% valid, 94.87% among definitive verdicts.
Edited and signed by
Stefan PerdalEditor and publisher89 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

