Teacher-built AI tools lifted exam scores in a 19-university trial

In a randomised trial across 55 applied science and business classes, students whose instructors designed their own AI tools did better on supervised, AI-free exams. Engagement and satisfaction did not measurably change. The study is a preprint and has not been peer reviewed.

6 min readIIBenchmarkedPreprint, not peer reviewed
Save
An artistically shot still life with a blue key inserted into a wooden block alongside a ring of brass keys.AI illustration
Illustration: AI generated

What the study shows

The authors ran a pre-registered randomised trial, meaning they published their plan before collecting results. It covered 55 applied STEM classes at 19 universities, mostly in statistics, analytics, operations and related business subjects. It ran over three semesters, from spring 2025 to spring 2026, with about 7,200 students enrolled.

Each instructor proposed their own AI tool for their own course. Classes were then randomly assigned to start using it straight away or one term later. Both groups received the same training and technical help, including a platform called Faculty Twin, built with a Google team, for setting up AI agents without code.

Learning was measured with proctored, in-person exams, mostly on paper, so students could not use AI while taking them. Each class's scores were compared with its own earlier years, and the change in AI classes was compared with the change in delayed classes. This is a difference-in-differences design. The authors report a gain of 0.32 standard deviations at student level and 0.26 at class level. Both results are statistically significant, but only just: the lower ends of their ranges sit close to zero.

The authors found no measurable change in engagement, such as attendance or logins, or in end-of-term course ratings. Those estimates were slightly negative but not statistically significant, and the ranges are wide.

The tools varied widely. The authors count 33 types across seven teaching functions. The most common were course navigation helpers (25 classes), tutors that ask guiding questions instead of giving answers (19) and practice tutors (18). Some classes used role-play, for example a simulated factory owner whose business problem students had to turn into a maths model.

In an end-of-course survey answered by about two in five students in AI classes, roughly 16 times as many said the tools improved their experience as said they worsened it. In interviews, about half of the 25 instructors reported uneven use: some students used the tools a lot and others barely at all.

Our reading

In our reading, the most valuable part of this study is the measurement. Many earlier studies of AI in education were short, covered one course, or tested students while AI was still available. Here the exams were AI-free and the trial ran over full terms in many institutions. That makes the learning result more credible than most in this area.

The headline explanation is less secure. The authors credit the positive result to instructors designing their own tools, and contrast it with earlier negative studies of generic chatbots. But the trial never compared teacher-built tools with a standard tool. The earlier studies also differ in age group, subject and measures. So the trial shows that this package helped: volunteer instructors, training, a support platform and their own designs. It does not show which part of the package mattered.

A practical example makes the point. A statistics lecturer who sets up a practice tutor that withholds final answers, uses the lecturer's own notation and generates fresh exam-style problems appears to give students something useful. Students said practice and exam preparation were what they valued most. Whether the same tool, built centrally and rolled out to every course, would work as well is an open question.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

II
Benchmarked
LevelStatusEvidence
I  Lab onlyMetThe work is a research study with a full methods description.
II  BenchmarkedMetMeasured on real proctored exams in a pre-registered randomised trial across 19 universities. This is a field evaluation, not a public benchmark, so it fits the level only loosely. Results are reported with several estimators.
III  Open sourceNot yetNo code, weights or tools are released. Analysis code is available to editors and referees on request. Restricted data are available only to qualified researchers under agreement. The Faculty Twin platform is not described as public.
IV  In productionNot yetThe tools were used in real courses during the trial and every control instructor adopted theirs afterwards. That is a research setting. There is no documented routine deployment beyond the study.

What would move it up
Releasing the analysis code, agent prompts or the Faculty Twin platform would reach level 3. Level 4 needs documented adoption by named institutions outside the study. Peer review, a replication outside applied STEM, a centrally designed comparison arm and repeat-term results would strengthen the claims.

The study

Jun Li, Robert Bray, Sébastien Martin et al.. Instructor-designed AI improves learning in a randomized trial across 19 universities. arXiv, 14 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY 4.0

Notes

  1. Li, Bray, Martin, Van den Steen, Wu et al. Instructor-designed AI improves learning in a randomized trial across 19 universities. arXiv preprint, 14 November 2026. CC BY 4.0. Source
  2. Main learning estimates: 0.32 SD at student level (SE 0.13, P=0.019, 95% CI 0.06 to 0.59, N=9,202 student observations including historical terms); 0.26 SD at class level (SE 0.12, P=0.038, 95% CI 0.02 to 0.51, N=101 class-terms).
  3. Engagement: -0.07 SD student level (P=0.60), -0.05 SD class level (P=0.60). Satisfaction: -0.11 SD student level (P=0.29), -0.10 SD class level (P=0.30).
  4. Analysis sample for learning: 44 of 67 experimental class-terms, 4,768 enrolled students. Mean consent rate for student-level data 79.4%.
  5. Pre-registered on the Open Science Framework on 28 February 2025, amended in March 2025, August 2025 and August 2026. The paper does not give a direct registration link. Source
  6. No code, prompts or platform released. Analysis code is available to editors and referees on request; restricted data to qualified researchers under agreement.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on In industry