AI coding agents can build few-label learning systems, but cannot reliably pick the best

A Cornell-led preprint finds coding agents can write programs that learn from a few labelled images plus many unlabelled ones, sometimes better than expert methods. Choosing the winning program without labels remains the hard part.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
A series of small, hand-whittled wooden keys in uneven rows on warm paper, alongside a single unattached blue tag.AI illustration
Illustration: AI generated

What the study shows

The study tests machine-learning engineering agents: AI systems that write, run and refine training code on their own. Until now they have mostly been tested on tasks with plenty of labelled data. This work tests them where labels are scarce but unlabelled examples are plentiful.

That setting is called semi-supervised learning: a model learns from a small labelled set plus a much larger unlabelled one. Many such methods exist, and each needs tuning to the dataset. The authors treat this as a search problem. An agent proposes, runs and improves candidate programs, guided by a weak feedback score.

Two feedback scores were tried. One holds back a few labelled images as a small, noisy test. The other uses label-free statistics, such as how well predictions form tidy clusters. The agents used the same vision models (CLIP and DINOv2) and the same fine-tuning tools as the expert baselines.

The tests covered six image datasets, from textures to eye scans to driving scenes, each with two label counts. That makes 12 settings, each run on three random data splits. The agent's best program beat the best of five published semi-supervised methods in 8 of the 12. The largest gain was about 22 points of accuracy on diabetic retinopathy images.

There is a catch. The best program was picked using test accuracy, which is unknown in practice. When the agent instead picked using its own feedback score, accuracy fell by as much as 47 percentage points against that best choice. On two datasets the feedback score barely tracked real accuracy. Expert methods selected without labels suffered similar losses.

The authors also tried spending some labels on choosing a program rather than on training. With labels split half and half, accuracy rose by about 20 to 26 points on retinopathy. It cost a little accuracy where label-free scores already worked.

They released ProgramSelectBench, a collection of 16,977 pre-computed agent and expert programs for testing selection methods. The work is a preprint and has not been peer reviewed.

Our reading

In our reading, the headline result is less about agents beating experts and more about where the difficulty moves. The agents were good at generating varied, sensible candidates. They often mixed several known techniques and adapted them to the data. On the retinopathy images, the best program trained no classifier at all. It compared images with labelled neighbours and respected the fact that disease grades are ordered.

But a pile of good candidates is only useful if the right one can be chosen. The paper shows that label-free scores can be badly misleading, and in ways that differ from one dataset to the next. The authors argue that stronger agents could widen this gap, because more candidates mean more chances to pick a poor one. We think that point deserves attention from anyone building automated machine-learning tools.

A practical example comes from the ecology tests. On a fish-detection task, the original expert labels took over 320 hours to draw. Agents given 10% of the labelled images scored about 42 to 43 on the standard detection measure. The expert pipeline scored 35.4 with the same 10% and 47.9 with all the data. That suggests a real saving in labelling, though not yet a full replacement. On an antelope task, the agents only matched the low-label expert baseline.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetLab-stage research prototype with controlled experiments on standard image benchmarks and two ecology datasets.
II  BenchmarkedMetResults on public benchmarks (VTAB-style DTD, SUN397, RESISC45, Retinopathy, CLEVR, KITTI) against six published baselines, with the caveats on oracle selection.
III  Open sourceMetProgramSelectBench (programs, predictions, unsupervised metrics, evaluation code) is released on Hugging Face, and the authors list a project website. The paper says analysis outputs and agent-generated programs will be released. The full AgentSSL search code is not clearly linked in the paper, so this level rests mainly on the benchmark data and evaluation code.
IV  In productionNot yetNo documented use in production by a named organisation. The ecology study uses real pipelines but is a case study, and agents fell short of expert baselines on BuckTales.

What would move it up
To reach level 4: a named team running agent-built SSL programs in a live pipeline, with a label-free way to pick the program that held up. Short of that: a clearly released search harness plus selection methods that cut regret on Retinopathy and KITTI-like cases, and results with more seeds and error bars.

Checked automatically every week, last on 16 November.

The study

Ethan Lin, Akanksha Sarkar, Ziang Liu et al.. AgentSSL: Can MLE Agents Leverage Unlabeled Data?. arXiv, 12 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY 4.0

Notes

  1. Lin, Sarkar, Liu, Branson and Sun. AgentSSL: Can MLE Agents Leverage Unlabeled Data? arXiv preprint 2611.17323, November 2026. Not peer reviewed. Licence CC BY 4.0. Source
  2. ProgramSelectBench dataset: 16,977 pre-computed programs across 35 selection tasks, with predictions, unsupervised metrics and evaluation code. Source
  3. Project website listed by the authors. Source
  4. Exact figures: largest oracle gain +21.81 points (Retinopathy, 40-shot); largest loss −13.92 (CLEVR, 20-shot); largest unsupervised selection regret −47.29 (AgentSSL with noisy validation, Retinopathy 80-shot). WildFin mAP@0.5:0.95: low-label expert 35.4, AgentSL 43.3, AgentSSL 42.0, full-data expert 47.9. BuckTales: low-label expert 28.1, AgentSL 27.7, AgentSSL 29.8, full-data expert 36.2.
  5. Compute: 2,484 GPU hours for all agent runs including re-evaluation on extra splits, against 7,628.7 GPU hours for tuning the baselines (Appendix Table 7).

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Agents