# Meta's AI agent improved a search ranker, lifting live searches 0.23%

Meta engineers let a Claude-driven agent propose, code, train and test changes to a search-suggestion model for a month. The best model raised search sessions by 0.23% in a live test, though humans steered throughout.

- Category: In industry
- Readiness level: IV (In production) on Science Underway's four-step scale, as proposed
- Published: 2026-11-19
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/meta-s-ai-agent-improved-a-search-ranker-lifting-live-searches-0-23

## The study

MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems (preprint, not peer reviewed). arXiv 2611.23910. https://arxiv.org/abs/2611.23910

## At a glance

- What it shows: An AI agent built at Meta ran over a thousand experiments on a ranking model behind search suggestions. Its best model slightly increased how often people searched in a live test against the existing model.
- What it does not show: It does not show that the agent works without people, that its design beats a better underlying model, or that the result repeats. Only one campaign, one model and one live test are reported.
- Why it matters: Most evidence for AI agents doing machine-learning work comes from benchmarks. This is a rare report of agent-made changes being tested on real users at large scale.

## What the study shows

Three Meta engineers built what they call machine learning engineer (MLE) agents. These are loops driven by a large language model, reported as Claude Opus 4.8. Each loop proposes a change to a ranking model, writes it as code, trains it, reads the results and decides whether to keep it.

The target was Nullstate, the model that ranks search suggestions shown before a user types anything. The authors say the system it serves handles over a billion requests a day. Each training run took about three and a half hours on 64 GPUs.

The second version of the agent ran for 31 days. It completed 1,097 candidate experiments and kept 325, about 30%. The first version, on the same model, kept about 16%. The authors stress this is not a fair comparison, because the scoring goal and search budget also changed.

Offline, on held-out data, the final model predicted both clicks and later engagement better than the starting model. The bigger gain was in later engagement, which the scoring goal weighted more heavily.

In a live A/B test of about a week, where some users got the old model and some the new one, search session volume rose by 0.23%. The 95% confidence interval was plus or minus 0.15 points. Other engagement measures were described only as moving in the right direction. The new model also slowed page rendering by an unstated amount.

The authors report that several kept changes were ideas the team had never tried, borrowed from published research. They also report that gains flattened early, and that later iterations added nothing. The work is a preprint for a NeurIPS workshop and has not been peer reviewed.

## Our reading

In our reading, the online test is the part that counts. A 0.23% lift sounds tiny, but on a system this large, small steady gains are what ranking teams chase. The interval is wide, though: the true effect could be as low as about 0.08%.

The paper is really about the scaffolding, or harness, around the language model. Three changes did most of the work, by the authors' account. They ran seven candidates at once to cope with hours-long training. They scored candidates on a measure chosen because it had tracked live results before. And they fed the agent a tidy summary, including a list of ideas that had already failed, instead of the full messy history.

That last fix is concrete. Malformed code from the agent fell from about a third of attempts to under 1%. For anyone running agents on long jobs, that is a useful, low-cost lesson. We rate the work level IV on our scale, proven in production, because Meta documents a live test on real users. It is a cautious IV: the paper does not clearly state a full launch, and no code is released.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/meta-s-ai-agent-improved-a-search-ranker-lifting-live-searches-0-23

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
