# AI coding agents can build few-label learning systems, but cannot reliably pick the best

A Cornell-led preprint finds coding agents can write programs that learn from a few labelled images plus many unlabelled ones, sometimes better than expert methods. Choosing the winning program without labels remains the hard part.

- Category: Agents
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-13
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/ai-coding-agents-can-build-few-label-learning-systems-but-cannot-reliably-pick-t

## The study

AgentSSL: Can MLE Agents Leverage Unlabeled Data? (preprint, not peer reviewed). arXiv 2611.17323. https://arxiv.org/abs/2611.17323

## At a glance

- What it shows: AI agents that write machine-learning code produced training programs that beat the best expert-designed method in 8 of 12 test settings. That held only when the researchers used the hidden test answers to pick the agent's best program.
- What it does not show: It does not show the agents win in everyday use. Without test answers to guide the choice, much of the gain disappears, and agents that ignored the unlabelled data did about as well in several checks.
- Why it matters: Labelling data is slow and costly in medicine, ecology and science. If agents can make use of cheap unlabelled data, the main remaining problem becomes knowing which of their many attempts to trust.

## What the study shows

The study tests machine-learning engineering agents: AI systems that write, run and refine training code on their own. Until now they have mostly been tested on tasks with plenty of labelled data. This work tests them where labels are scarce but unlabelled examples are plentiful.

That setting is called semi-supervised learning: a model learns from a small labelled set plus a much larger unlabelled one. Many such methods exist, and each needs tuning to the dataset. The authors treat this as a search problem. An agent proposes, runs and improves candidate programs, guided by a weak feedback score.

Two feedback scores were tried. One holds back a few labelled images as a small, noisy test. The other uses label-free statistics, such as how well predictions form tidy clusters. The agents used the same vision models (CLIP and DINOv2) and the same fine-tuning tools as the expert baselines.

The tests covered six image datasets, from textures to eye scans to driving scenes, each with two label counts. That makes 12 settings, each run on three random data splits. The agent's best program beat the best of five published semi-supervised methods in 8 of the 12. The largest gain was about 22 points of accuracy on diabetic retinopathy images.

There is a catch. The best program was picked using test accuracy, which is unknown in practice. When the agent instead picked using its own feedback score, accuracy fell by as much as 47 percentage points against that best choice. On two datasets the feedback score barely tracked real accuracy. Expert methods selected without labels suffered similar losses.

The authors also tried spending some labels on choosing a program rather than on training. With labels split half and half, accuracy rose by about 20 to 26 points on retinopathy. It cost a little accuracy where label-free scores already worked.

They released ProgramSelectBench, a collection of 16,977 pre-computed agent and expert programs for testing selection methods. The work is a preprint and has not been peer reviewed.

## Our reading

In our reading, the headline result is less about agents beating experts and more about where the difficulty moves. The agents were good at generating varied, sensible candidates. They often mixed several known techniques and adapted them to the data. On the retinopathy images, the best program trained no classifier at all. It compared images with labelled neighbours and respected the fact that disease grades are ordered.

But a pile of good candidates is only useful if the right one can be chosen. The paper shows that label-free scores can be badly misleading, and in ways that differ from one dataset to the next. The authors argue that stronger agents could widen this gap, because more candidates mean more chances to pick a poor one. We think that point deserves attention from anyone building automated machine-learning tools.

A practical example comes from the ecology tests. On a fish-detection task, the original expert labels took over 320 hours to draw. Agents given 10% of the labelled images scored about 42 to 43 on the standard detection measure. The expert pipeline scored 35.4 with the same 10% and 47.9 with all the data. That suggests a real saving in labelling, though not yet a full replacement. On an antelope task, the agents only matched the low-label expert baseline.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/ai-coding-agents-can-build-few-label-learning-systems-but-cannot-reliably-pick-t

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
