True posts, false conclusions: AI agents fill gaps with rumours
A preprint shows that language-model agents can be steered towards rumours using only truthful posts, arranged to hint at a link that none of them states. A simple support check helps, but only partly.


What the study shows
The researchers, from the University of Liverpool and MBZUAI, study what they call narrative overfitting. This is when an AI agent links separate facts with a relationship it has assumed, then treats that assumption as established. The work is a preprint and has not been peer reviewed.
To test it, they built an attack pipeline called Generative Montage. Three AI roles, all run on GPT-4.1-mini, pick true posts from real Twitter coverage, cut them into short posts and reorder them. The aim is to suggest a rumour without ever stating it. A second AI role checks that every post stays factually accurate.
They ran this on 1,090 rumour claims from six 2014 to 2015 news events, including the Charlie Hebdo attack and the Sydney siege. Fourteen models from the GPT, Claude, Qwen and DeepSeek families acted as analysts. Twelve of them accepted the rumours in more than 65% of runs. The most resistant, Claude-4.5-Haiku, still accepted about four in ten.
One example: posts mentioned hostages in a Sydney cafe and, separately, that 13 people had been counted. The analyst concluded there were 13 hostages. Nothing in the feed linked the two.
Group checks did not solve the problem. A majority vote across five analysts still accepted rumours in roughly 60% to 75% of cases, depending on model family. An AI judge that could see the original posts and all five reports accepted them in roughly 58% to 70% of cases.
The authors then built a proof-of-concept safeguard, Grounded Monitor. It labels each post by whether it directly establishes the claim or merely makes it plausible. If a feed pushes hard towards a claim but no single post establishes it, the monitor raises a warning. Given that warning plus an 'uncertain' answer option, four small open models made more correct accept-or-reject decisions, by about 11 to 29 percentage points on the original feeds.
Our reading
In our reading, the most useful contribution is the framing. Most defences against AI misinformation check whether individual statements are true. This study shows the error can live entirely in the gaps between true statements. A fact-checker that verifies each sentence would pass every post in these attack feeds.
The finding about review is the one that should worry system designers most. Five analysts reading the same feed are not five independent sources. Their agreement looks like corroboration but adds no new evidence. The authors report that a reviewing AI with access to the original posts still let most rumours through. We note, though, that the judge was a fairly small open model, and stronger judges were not tested.
On our four-level readiness scale, we place this at level III, open source available. The paper reports results on a public dataset and links code and data on GitHub, though we have not checked the repository's contents or licence. There is no documented use by any organisation, and the authors themselves call the monitor a proof of concept.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | Controlled experiments in a simulated social feed, with a proof-of-concept monitor. No live deployment. |
| II Benchmarked | Met | Results are reported on a public dataset (PHEME/CoPHEME, 1,090 targets, six events) across 14 named models, with ablations, confidence intervals and controls. The monitor is tested on 238 targets in two settings. The paper builds its own benchmark and attack set, so this is not an independent leaderboard. |
| III Open source | Met | The paper gives a GitHub link for both code and datasets (github.com/CharlesJW222/Narrative-Overfitting-NMI). We could not inspect the repository or its licence, so completeness is unverified. |
| IV In production | Not yet | No documented use by a named organisation. The monitor is described as proof of concept, and no deployment or field test is reported. |
What would move it up
To reach level 4: a named organisation running the monitor, or a similar support check, in a live summarisation or fact-checking product, with reported false-alarm and miss rates. Before that, independent replication using the released code, tests on current frontier models and fresh events, and a comparison against simple baselines such as an 'uncertain' option alone would strengthen the case.
The study
Jinwei Hu, Qisong He, Xinmiao Huang et al.. Language-model agents' beliefs can be manipulated via truthful evidence in open channel. arXiv, 16 November 2026. Read the paper
Notes
- Hu, He, Huang, Sun, Dong and Huang, 'Language-model agents' beliefs can be manipulated via truthful evidence in open channel', arXiv preprint, 16 November 2026, CC BY 4.0. Source
- Code and data, as linked in the paper. We have not inspected the repository or its licence. Source
- Overall rumour acceptance ranged from 91.5% (Claude-3-Haiku) to 42.4% (Claude-4.5-Haiku). Mean acceptance was 74.4% for proprietary and 70.3% for open-weight models (Fig. 1a, Extended Data Table 1).
- Downstream acceptance: majority vote 59.6% to 75.0%; DeepSeek-R1-Distill-Qwen-14B judge 57.7% to 69.5%, by model family (Fig. 1c).
- Grounded Monitor alarm: 91.1% of attack feeds and 4.4% of chronological control feeds. Decision accuracy gains: 11.2 to 29.0 points (original), 5.6 to 13.7 points (anonymised); e.g. Qwen2.5-14B rose from 47.0% to 75.1% (Table 2, Fig. 5).
- Salience AUROC between rumour and true-claim montages was 0.502; grounding was found in 38.8% of true-claim feeds and 6.3% of rumour feeds (Table 2).
- Chain-of-thought raised acceptance by 3.1 points (Qwen2.5-7B) and 6.8 points (DeepSeek-R1-7B) on Charlie Hebdo (Extended Data Table 3).
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections

