# Which models and sampling choices matter when humans and AI count together

A preprint benchmark of 12 image-counting tasks finds that free, off-the-shelf vision-language models are a strong starting point for estimating totals with a small human-labelled sample. Letting the model pick which images people check usually helps further.

- Category: Vision & multimodal
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-12
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/which-models-and-sampling-choices-matter-when-humans-and-ai-count-together

## The study

What Factors Matter for Collection-Level Counting with Human-AI Collaboration? (preprint, not peer reviewed). DOI 10.48550/arxiv.2611.14380, arXiv 2611.14380. https://arxiv.org/abs/2611.14380

## At a glance

- What it shows: Across 12 counting tasks, combining model predictions with a small human-labelled sample beat both model-only counts and labels-only estimates in most cases. Open-source multimodal models did best at small labelling budgets, and fine-tuned detectors took over at larger ones.
- What it does not show: It does not show operational use, multi-class counting or real cost savings. Its confidence intervals also cover the true total less often than the nominal 95%.
- Why it matters: Many scientific and policy totals, from infected cells to wildlife populations, come from image collections too large to label by hand. This work gives practical rules for choosing a model and a sampling method before labelling starts.

## What the study shows

The problem is estimating the total number of objects, such as cells, birds or ships, across a large image collection. Full human labelling is too slow and raw model counts can be biased, especially when images differ from a model's training data. The approach studied here has a model predict a count for every image tile, then has humans label a sample of tiles. A statistical estimator uses those labels to correct the model's total. The estimators are unbiased, meaning they are right on average however good the model is. A better model just means fewer labels are needed for a given precision.

The authors, from the University of Massachusetts Amherst and the University of Wyoming, built a benchmark of 12 single-class tasks from four public datasets: malaria microscopy, ground-level photos of bird flocks, aerial wildlife surveys in Kenya, and the DOTA satellite and aerial imagery set. Each task has between 347 and 5,176 image units. The tasks are grouped by how many units are empty, from under 5% to 90% or more. The paper is a preprint and has not been peer reviewed.

They compared 13 models. These were two Faster R-CNN detectors fine-tuned on the target data as labels arrive, and three open-vocabulary detectors, which find objects named in text. The rest were five open-source and three commercial multimodal large language models (MLLMs, models that read images and answer questions in text). MLLMs were simply asked how many objects were in each image. Each model was paired with several estimators and scored by its absolute fractional error, the gap between estimate and true total as a share of the true total, averaged over 1,000 trials.

The authors report that model-assisted estimators beat both trusting the model alone and a model-free Monte Carlo estimate, which scales up the average of the human-labelled units. Averaged over the 12 tasks, the best open-source MLLM was under 10% error at 1% of units labelled, where Monte Carlo was 3.8 to 6 times worse. Open-source MLLMs had the lowest error from 1% to about 30% labelled. Fine-tuned detectors improved fastest and overtook every frozen model family at about 40%. At 5% labelled, open-source MLLMs won 9 of the 11 tasks that had a unique winner. At 50%, detectors won 8. InternVL3.5-8B won the most settings, 34 of 120. Of the commercial models, only Qwen3.7-Plus won any setting, and the open-vocabulary detectors never won.

Importance sampling, where humans label the units the model predicts hold the most objects, won 63% of 1,560 model, task and budget cases. Monte Carlo won 14.6%, mostly on dense tasks. The advantage of importance sampling grew as targets became rarer. Choosing well also matters because MLLM totals varied wildly, from 0.004 to 55 times the true count. A pilot of 50 labelled units, ranking frozen models by Hellinger distance (a measure of how closely the spread of predicted counts across units matches the true spread), picked a model whose median error was 1.04 to 1.06 times that of the best candidate.

In a case study with the Wyoming Game and Fish Department, the method was applied to 6,129 aerial images of pronghorn, 98.8% of which contained no animals. The true count was 711. A detector pre-trained on four earlier surveys, plus a small model that ranks tiles for review, reached the survey's ±5% uncertainty target after a median of 3.2% of imagery reviewed over 100 runs. That is about 200 full images. Four runs needed 27 to 28.5%. In 93 of the 100 runs the final 95% interval contained the true count.

## Our reading

In our reading, the most useful finding is practical rather than statistical: a general-purpose open-source model, asked a plain question, is good enough to steer human labelling on most of these tasks. No training is needed. Model choice still matters a great deal, and the 50-unit pilot test gives a cheap way to make that choice. The pilot result also shows that ranking models by how well their counts match across images works better than ranking by how close their total is. That is a useful correction to the instinct to pick the model with the best headline count.

Readiness: we rate this level II, Benchmarked. The results come from public datasets with exhaustive ground truth, and humans are simulated by revealing known labels. The paper text links no code or benchmark files, so level III is not met. The pronghorn study involved a named agency and real survey imagery, but it is a retrospective test against a full manual count, not documented production use, so level IV is not met either.

The headline curves need careful reading. Each model family is represented by its best model and estimator at each budget, which a practitioner would not know in advance. The pilot procedure narrows that gap for frozen models only. The intervals are also optimistic: across the benchmark, 95% intervals contained the truth 89% of the time on average, and only 82% at a 5% budget.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/which-models-and-sampling-choices-matter-when-humans-and-ai-count-together

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
