# Teacher-built AI tools lifted exam scores in a 19-university trial

In a randomised trial across 55 applied science and business classes, students whose instructors designed their own AI tools did better on supervised, AI-free exams. Engagement and satisfaction did not measurably change. The study is a preprint and has not been peer reviewed.

- Category: In industry
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-17
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/teacher-built-ai-tools-lifted-exam-scores-in-a-19-university-trial

## The study

Instructor-designed AI improves learning in a randomized trial across 19 universities (preprint, not peer reviewed). arXiv 2611.19099. https://arxiv.org/abs/2611.19099

## At a glance

- What it shows: When university teachers built their own AI study aids for their courses, students scored higher on supervised exams taken without AI. The gain was moderate and measured across about 4,800 students in 44 course runs.
- What it does not show: It does not show that letting teachers design the tools is the reason for the gain, because no class used a standard, centrally designed tool for comparison. It also does not show which kinds of tool work best.
- Why it matters: Several recent studies found that handing students a general chatbot can hurt learning. This trial suggests that AI shaped by the people who know the course can help instead.

## What the study shows

The authors ran a pre-registered randomised trial, meaning they published their plan before collecting results. It covered 55 applied STEM classes at 19 universities, mostly in statistics, analytics, operations and related business subjects. It ran over three semesters, from spring 2025 to spring 2026, with about 7,200 students enrolled.

Each instructor proposed their own AI tool for their own course. Classes were then randomly assigned to start using it straight away or one term later. Both groups received the same training and technical help, including a platform called Faculty Twin, built with a Google team, for setting up AI agents without code.

Learning was measured with proctored, in-person exams, mostly on paper, so students could not use AI while taking them. Each class's scores were compared with its own earlier years, and the change in AI classes was compared with the change in delayed classes. This is a difference-in-differences design. The authors report a gain of 0.32 standard deviations at student level and 0.26 at class level. Both results are statistically significant, but only just: the lower ends of their ranges sit close to zero.

The authors found no measurable change in engagement, such as attendance or logins, or in end-of-term course ratings. Those estimates were slightly negative but not statistically significant, and the ranges are wide.

The tools varied widely. The authors count 33 types across seven teaching functions. The most common were course navigation helpers (25 classes), tutors that ask guiding questions instead of giving answers (19) and practice tutors (18). Some classes used role-play, for example a simulated factory owner whose business problem students had to turn into a maths model.

In an end-of-course survey answered by about two in five students in AI classes, roughly 16 times as many said the tools improved their experience as said they worsened it. In interviews, about half of the 25 instructors reported uneven use: some students used the tools a lot and others barely at all.

## Our reading

In our reading, the most valuable part of this study is the measurement. Many earlier studies of AI in education were short, covered one course, or tested students while AI was still available. Here the exams were AI-free and the trial ran over full terms in many institutions. That makes the learning result more credible than most in this area.

The headline explanation is less secure. The authors credit the positive result to instructors designing their own tools, and contrast it with earlier negative studies of generic chatbots. But the trial never compared teacher-built tools with a standard tool. The earlier studies also differ in age group, subject and measures. So the trial shows that this package helped: volunteer instructors, training, a support platform and their own designs. It does not show which part of the package mattered.

A practical example makes the point. A statistics lecturer who sets up a practice tutor that withholds final answers, uses the lecturer's own notation and generates fresh exam-style problems appears to give students something useful. Students said practice and exam preparation were what they valued most. Whether the same tool, built centrally and rolled out to every course, would work as well is an open question.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/teacher-built-ai-tools-lifted-exam-scores-in-a-19-university-trial

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
