Which models and sampling choices matter when humans and AI count together
A preprint benchmark of 12 image-counting tasks finds that free, off-the-shelf vision-language models are a strong starting point for estimating totals with a small human-labelled sample. Letting the model pick which images people check usually helps further.

AI illustrationWhat the study shows
The problem is estimating the total number of objects, such as cells, birds or ships, across a large image collection. Full human labelling is too slow and raw model counts can be biased, especially when images differ from a model's training data. The approach studied here has a model predict a count for every image tile, then has humans label a sample of tiles. A statistical estimator uses those labels to correct the model's total. The estimators are unbiased, meaning they are right on average however good the model is. A better model just means fewer labels are needed for a given precision.
The authors, from the University of Massachusetts Amherst and the University of Wyoming, built a benchmark of 12 single-class tasks from four public datasets: malaria microscopy, ground-level photos of bird flocks, aerial wildlife surveys in Kenya, and the DOTA satellite and aerial imagery set. Each task has between 347 and 5,176 image units. The tasks are grouped by how many units are empty, from under 5% to 90% or more. The paper is a preprint and has not been peer reviewed.
They compared 13 models. These were two Faster R-CNN detectors fine-tuned on the target data as labels arrive, and three open-vocabulary detectors, which find objects named in text. The rest were five open-source and three commercial multimodal large language models (MLLMs, models that read images and answer questions in text). MLLMs were simply asked how many objects were in each image. Each model was paired with several estimators and scored by its absolute fractional error, the gap between estimate and true total as a share of the true total, averaged over 1,000 trials.
The authors report that model-assisted estimators beat both trusting the model alone and a model-free Monte Carlo estimate, which scales up the average of the human-labelled units. Averaged over the 12 tasks, the best open-source MLLM was under 10% error at 1% of units labelled, where Monte Carlo was 3.8 to 6 times worse. Open-source MLLMs had the lowest error from 1% to about 30% labelled. Fine-tuned detectors improved fastest and overtook every frozen model family at about 40%. At 5% labelled, open-source MLLMs won 9 of the 11 tasks that had a unique winner. At 50%, detectors won 8. InternVL3.5-8B won the most settings, 34 of 120. Of the commercial models, only Qwen3.7-Plus won any setting, and the open-vocabulary detectors never won.
Importance sampling, where humans label the units the model predicts hold the most objects, won 63% of 1,560 model, task and budget cases. Monte Carlo won 14.6%, mostly on dense tasks. The advantage of importance sampling grew as targets became rarer. Choosing well also matters because MLLM totals varied wildly, from 0.004 to 55 times the true count. A pilot of 50 labelled units, ranking frozen models by Hellinger distance (a measure of how closely the spread of predicted counts across units matches the true spread), picked a model whose median error was 1.04 to 1.06 times that of the best candidate.
In a case study with the Wyoming Game and Fish Department, the method was applied to 6,129 aerial images of pronghorn, 98.8% of which contained no animals. The true count was 711. A detector pre-trained on four earlier surveys, plus a small model that ranks tiles for review, reached the survey's ±5% uncertainty target after a median of 3.2% of imagery reviewed over 100 runs. That is about 200 full images. Four runs needed 27 to 28.5%. In 93 of the 100 runs the final 95% interval contained the true count.
Our reading
In our reading, the most useful finding is practical rather than statistical: a general-purpose open-source model, asked a plain question, is good enough to steer human labelling on most of these tasks. No training is needed. Model choice still matters a great deal, and the 50-unit pilot test gives a cheap way to make that choice. The pilot result also shows that ranking models by how well their counts match across images works better than ranking by how close their total is. That is a useful correction to the instinct to pick the model with the best headline count.
Readiness: we rate this level II, Benchmarked. The results come from public datasets with exhaustive ground truth, and humans are simulated by revealing known labels. The paper text links no code or benchmark files, so level III is not met. The pronghorn study involved a named agency and real survey imagery, but it is a retrospective test against a full manual count, not documented production use, so level IV is not met either.
The headline curves need careful reading. Each model family is represented by its best model and estimator at each budget, which a practitioner would not know in advance. The pilot procedure narrows that gap for frozen models only. The intervals are also optimistic: across the benchmark, 95% intervals contained the truth 89% of the time on average, and only 82% at a 5% budget.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The method works in a lab setting: a controlled benchmark with exhaustive ground truth and simulated human labelling. |
| II Benchmarked | Met | Results across 12 tasks drawn from four public datasets (BBBC041 malaria, Birds, Kenyan wildlife, DOTA v1.5), comparing 13 models and 9 estimators against model-only and Monte Carlo baselines, averaged over 1,000 trials. |
| III Open source | Not yet | The paper text gives no link to released code, weights or benchmark construction scripts. The models and datasets used are public, but the authors' own pipeline is not shown as released. |
| IV In production | Not yet | The pronghorn study was done with the Wyoming Game and Fish Department on real imagery, but it is a case study with exhaustive ground truth available. The paper does not document it being used to produce an official population estimate. |
What would move it up
Release the code, benchmark task definitions and tiling lists to reach level 3. For level 4, the paper would need documented operational use, for example Wyoming Game and Fish using the method to produce an official survey estimate without exhaustive annotation. Better interval coverage and multi-class tests would also strengthen the case.
Checked automatically every week, last on 16 November.
The study
Seoyun Jeong, Logan Lawrence, Owen L. Martin et al.. What Factors Matter for Collection-Level Counting with Human-AI Collaboration?. arXiv, 10 November 2026. doi:10.48550/arxiv.2611.14380
Notes
- Jeong, Lawrence, Martin, Koger, Sheldon and Maji. What Factors Matter for Collection-Level Counting with Human-AI Collaboration? arXiv preprint 2611.14380, November 2026. Not peer reviewed. Licence CC BY-NC-SA. Source
- No code or data release is linked in the paper text.
- The framework builds on Hamilton et al., Active measurement: Efficient estimation at scale, NeurIPS 38 (2026b).
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections


