Meta's AI agent improved a search ranker, lifting live searches 0.23%

Meta engineers let a Claude-driven agent propose, code, train and test changes to a search-suggestion model for a month. The best model raised search sessions by 0.23% in a live test, though humans steered throughout.

5 min readIVIn productionPreprint, not peer reviewed
Save
A high-angle studio photograph of a long row of small wooden blocks on warm paper, with a few blocks scattered forward and a single deep blue block near the start.AI illustration
Illustration: AI generated

What the study shows

Three Meta engineers built what they call machine learning engineer (MLE) agents. These are loops driven by a large language model, reported as Claude Opus 4.8. Each loop proposes a change to a ranking model, writes it as code, trains it, reads the results and decides whether to keep it.

The target was Nullstate, the model that ranks search suggestions shown before a user types anything. The authors say the system it serves handles over a billion requests a day. Each training run took about three and a half hours on 64 GPUs.

The second version of the agent ran for 31 days. It completed 1,097 candidate experiments and kept 325, about 30%. The first version, on the same model, kept about 16%. The authors stress this is not a fair comparison, because the scoring goal and search budget also changed.

Offline, on held-out data, the final model predicted both clicks and later engagement better than the starting model. The bigger gain was in later engagement, which the scoring goal weighted more heavily.

In a live A/B test of about a week, where some users got the old model and some the new one, search session volume rose by 0.23%. The 95% confidence interval was plus or minus 0.15 points. Other engagement measures were described only as moving in the right direction. The new model also slowed page rendering by an unstated amount.

The authors report that several kept changes were ideas the team had never tried, borrowed from published research. They also report that gains flattened early, and that later iterations added nothing. The work is a preprint for a NeurIPS workshop and has not been peer reviewed.

Our reading

In our reading, the online test is the part that counts. A 0.23% lift sounds tiny, but on a system this large, small steady gains are what ranking teams chase. The interval is wide, though: the true effect could be as low as about 0.08%.

The paper is really about the scaffolding, or harness, around the language model. Three changes did most of the work, by the authors' account. They ran seven candidates at once to cope with hours-long training. They scored candidates on a measure chosen because it had tracked live results before. And they fed the agent a tidy summary, including a list of ideas that had already failed, instead of the full messy history.

That last fix is concrete. Malformed code from the agent fell from about a third of attempts to under 1%. For anyone running agents on long jobs, that is a useful, low-cost lesson. We rate the work level IV on our scale, proven in production, because Meta documents a live test on real users. It is a cautious IV: the paper does not clearly state a full launch, and no code is released.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

IV
In production
LevelStatusEvidence
I  Lab onlyMetThe method was built and run on a real system, with a clear description of the loop, objective and context design.
II  BenchmarkedMetOffline results are reported against a baseline ranker (weighted NE 0.6611 to 0.6388). Strictly, this is an internal evaluation set rather than a public benchmark, so this level is only loosely met. The paper also compares against no external benchmark or other agent systems.
III  Open sourceNot yetNo code, weights, data or link is released. The paper answers 'No' to open access for code and data, and describes the method only in prose.
IV  In productionMetMeta reports that the agent-discovered model was served in a live A/B test on a large search platform (over a billion requests a day), with a statistically significant +0.23% search volume. This is a test arm, not a stated full launch, so full production deployment is not documented.

What would move it up
Level 3 evidence needs released code, harness prompts or at least a reference implementation that others can run. For firmer level 4 evidence, the authors should state that the model launched fully, and report repeat runs, the latency cost, and downstream metrics with numbers. A controlled ablation (same objective and budget, varying only the harness or the base model) would also strengthen the main claim.

The study

Matthew Li, Albert Liang, Dilip Raj. MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems. arXiv, 18 November 2026. Read the paper

Preprint, not peer reviewed · Licence arXiv (no reuse)

Notes

  1. Li, Liang and Raj, MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems. arXiv preprint, NeurIPS 2026 Workshop for Autonomous ML Research. Not peer reviewed. Source
  2. No code, data or weights are released.
  3. Exact figures: 1,097 completed candidate runs, 325 accepted (29.6%), 131 iterations, 155 variance probes. Weighted normalised entropy fell from 0.6611 to 0.6388 (minus 0.0224). Version 1.0 on Nullstate: 10 of 64 kept (15.6%). Malformed output fell from 33.8% to 0.8%. Online: search session volume +0.23% ± 0.15, 95% CI by Fieller's method. Compute about 360,000 GPU-hours over 31 days; median run 3.54 hours on 64 GPUs.
  4. Closest related work cited: Wang et al., Self-Evolving Recommendation System, arXiv:2602.10226 (YouTube). Source

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on In industry