On AgentDojo, a taxonomy-guided attacker finds more kinds of prompt injection
A new red-teaming method for AI agents tries every catalogued way of planting hidden instructions. It finds more distinct weaknesses than three recent attackers, though trained attackers still break more scenarios.

AI illustrationWhat the study shows
AI agents read emails, web pages and tool outputs that neither the user nor the developer wrote. Prompt injection hides instructions in that content, so the agent carries out the attacker's wishes. Red teaming, where testers attack a system to find its weak points, is the main way to measure this risk.
Most automated red teamers aim to maximise attack success rate (ASR), the share of test scenarios they manage to break. The authors argue this hides something. An attacker can score well by reusing one or two tricks, leaving other techniques untested.
Their method, AutoPI, works from PIAtlas, a catalogue of 132 injection sub-techniques grouped into 34 techniques and six classes. The same authors published the catalogue earlier. For each scenario, an attacker model writes one short injection per sub-technique. A second model then diagnoses failed attempts, and the attacker reworks them. The method only sees the target's outputs, needs no training and plugs into existing benchmarks.
The authors tested it on AgentDojo, a standard agent benchmark, and IPI Arena, a newer one used in vendor reports. The targets were the open model Qwen3.6-35B and the hosted Claude Haiku 5.5. They compared it with three published attackers (PIMiner, Climbing the Hill and AutoDojo) and with AgentDojo's own fixed attack.
The authors report that on AgentDojo against Qwen3.6-35B, AutoPI's successful attacks covered 31 of 34 techniques. The best rival covered 22. A separate AutoPI setting, which reworks AgentDojo's fixed attack, broke about 68% of scenarios at equal compute. That beat PIMiner but trailed Climbing the Hill, at about 84%. Both of those rivals are trained before the attack; AutoPI is not.
Against Claude Haiku 5.5, every attacker broke only four to six of 97 scenarios. AutoPI broke six. On IPI Arena it broke 14 of 41, slightly fewer than the two trained rivals. This is a preprint and has not been peer reviewed.
Our reading
In our reading, the useful idea here is the yardstick more than the attacker. A security score that says an agent resists 90% of attacks says little if those attacks all use the same trick. Counting how many distinct techniques still work is a more honest summary of a worst-case risk.
The evidence for broader coverage on AgentDojo is reasonably solid. The lead held when attackers were compared at the same number of successes, and every attacker's output was labelled by the same procedure. The evidence for higher success rates is mixed. AutoPI leads only on Haiku, where the margins are a scenario or two, and trails trained attackers elsewhere.
A practical example: a team testing an email agent might find that its defences stop authority-style attacks. A sweep like this could still show the agent obeying an instruction written in French, or framed as fiction. Both broke scenarios here that no rival broke.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The method is described and tested in a research setting on two agent benchmarks. |
| II Benchmarked | Met | Results on public benchmarks (AgentDojo v1.2.2, IPI Arena) with three published adaptive attackers (PIMiner, CtH, AutoDojo) and a static-seed baseline, at equal cost and in a limit test. |
| III Open source | Met | The authors state they release the engine, adapters, configurations and generated attack corpus (github.com/stevendamianakis/autopi). The release was not inspected from the paper, and the licence is not stated in the paper. |
| IV In production | Not yet | No documented use by a named organisation in production. All evidence is from research benchmarks against off-the-shelf models with no deployed system. |
What would move it up
Level 4 needs documented use by a named organisation, for example a vendor or security team using AutoPI to test a deployed agent or to produce adversarial-training data. Evidence that training on its corpus makes a model measurably more robust, with independent replication on stronger targets and other generator models, would also strengthen the case.
The study
Stavros Damianakis, Konstantinos Psounis. AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM Agents. arXiv, 18 November 2026. Read the paper
Notes
- Damianakis and Psounis, AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM Agents, arXiv preprint, November 2026. Source
- Code, adapters and generated attack corpus, as stated by the authors. Not inspected; licence not stated in the paper. Source
- Project site. Source
- Exact figures on AgentDojo against Qwen3.6-35B: AutoPI Full covered 31/34 techniques and 78/132 sub-techniques; best baseline 22/34 and 48/132 (PIMiner). At 98 successes each: 22.9 techniques for AutoPI Full, 20.4 PIMiner, 16.5 CtH. ASR at equal cost: AutoPI Seed-Exploit 68.2%, PIMiner 57.3%, CtH 84.1%, AutoPI Full 42.9%, static seed 10.4%.
- Haiku 5.5: AutoPI Full broke 6 of 97 scenarios, CtH 5, PIMiner 5, static seed 4; AutoDojo 4 of 57 it ran. IPI Arena: AutoPI Full 14 of 41, PIMiner 16, CtH 15.
- Cost per attempt with training counted: AutoPI about 0.6 GPU-seconds, AutoDojo 2.9, CtH at least 4.7, PIMiner about 14.5. Replayed AutoPI injections succeeded against Progent, DRIFT and ROPE in 7.7%, 2.6% and 8.3% of 351 scenarios.
- Readiness: level III (open source available), based on public benchmark results and the stated code release.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

