Small Qwen3.5 model picks the evidence, cutting long-document wait up to 11.6x

A tiny vision-language model skims long image-and-text inputs and hands a larger model only the pages, images and paragraphs that matter. On MMLongBench, three larger models started answering several times sooner, with average accuracy held or slightly raised.

7 min readIIBenchmarkedPreprint, not peer reviewed
Save
A studio still-life photograph of a tall, untidy stack of loose papers and small photo prints, with three neatly aligned sheets clipped together by a single deep blue paper clip on an off-white background.AI illustration
Illustration: AI generated

What the study shows

Vision-language models answer questions about images and text together. When the input is long, such as a report of many pages or a few hundred photos, they are slow to start. Most of that wait is the prefill: the step where the model reads the whole input before writing anything. The authors report that on Qwen3-VL-32B, an input of about 105,000 tokens (chunks of text or image) waited about 19 seconds for the first output word. Almost all of it was prefill.

The method, which the authors call SkimPrefill, adds a small selector model in front. The selector is Qwen3.5-0.8B, a hybrid model in which most layers take time that grows only in step with input length. It reads the whole input with images at half resolution, but stops after 8 of its 24 layers. One attention head there, a part of the model that weighs how strongly the question relates to each piece of the input, scores every image and every paragraph. The top-scoring units, about 10% of the tokens, go to the larger target model as an ordinary, shorter prompt. The head was chosen on 50 synthetic documents that share nothing with the test data. No training was needed.

The authors tested on MMLongBench at its longest setting, about 128,000 tokens: 1,750 questions from 19 datasets across five task types. The targets were Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B. They compared against four published pruning methods that need no training: FastV, FlexSelect, SGP and SpecPrefill. All kept the same 10% of tokens.

The authors report that time to first token fell by 3.7 times on Qwen3.5-9B, 11.6 times on Gemma-4-12B and 8.2 times on Qwen3-VL-32B. Average accuracy rose slightly on the two Qwen targets and by about 10 points on Gemma-4-12B. SkimPrefill had the best average of the pruning methods on all three targets. Most rivals gave little or no speed gain on two of the targets, and the fastest rival, FastV, lost more than 20 points.

The gains were uneven. Needle-in-a-haystack questions, where the answer sits in a few images or sentences, rose by 11 to 26 points. Summarisation, which needs the whole document, fell by 3 to 4 points. On one long-document dataset, LongDocURL, the selector kept no answer page at all for 18 of 100 questions.

The targets, which reason step by step before answering, also wrote shorter reasoning with SkimPrefill. On Qwen3.5-9B, average output fell from about 6,400 to 3,500 tokens. The full time to a finished answer was 2.2 to 3.8 times shorter. This is a preprint and has not been peer reviewed. On our readiness scale it sits at level II, Benchmarked: public benchmark results exist, but no code has been released yet.

Our reading

In our reading, the most useful finding is about what to keep, not how to score it. Earlier methods kept the most important single tokens. On long mixed inputs those tokens end up scattered: a few patches of most images and fragments of most paragraphs. The authors show this breaks the evidence apart. When they kept the top single tokens using SkimPrefill's own scores, accuracy collapsed on both targets tested. Keeping whole images and paragraphs with the same scores did not. Whole units also raised the accuracy of two rival scoring methods.

The headline speed figures need context. The largest gain, on Gemma-4-12B, partly reflects a weak starting point. With the full input, Gemma often hit its 32,000-token reasoning limit and scored close to zero on one task type, many-shot in-context learning. The authors report that Gemma scored higher there the less input it read. So the 10-point gain on Gemma says as much about Gemma struggling with very long inputs as about the selector. The Qwen results, a 2 to 3 point gain at 3.7 to 8.2 times faster, are the more representative picture.

A concrete case: an assistant answering questions over a bundle of scanned product manuals, or a few hundred site-inspection photos. If the answer sits in a handful of pages or pictures, this approach could cut a 20-second pause to a few seconds. If the user asks for a summary of the whole bundle, it would likely do worse than reading everything.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

II
Benchmarked
LevelStatusEvidence
I  Lab onlyMetA working method with a clear design and experiments, but also the starting point for all readiness levels.
II  BenchmarkedMetResults on the public MMLongBench benchmark at 128K tokens across 19 datasets and three target models, compared against four published pruning baselines, with ablations and timing.
III  Open sourceNot yetThe paper says code and scripts 'will' be released. No link or released code, weights or data is given in the paper.
IV  In productionNot yetNo documented use in production by a named organisation.

What would move it up
Releasing the code and calibration scripts publicly would reach level 3. Level 4 would need documented deployment by a named organisation. Independent replication on other benchmarks, other target families and shorter, more varied inputs would also strengthen the case.

The study

Xiaolin Lin, Kai-Chiang Wu, Bing Li et al.. Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs. arXiv, 18 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY NC SA 4.0

Notes

  1. Lin, Wu, Li and Zhang, 'Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs', arXiv preprint, November 2026, not peer reviewed. Licence CC BY-NC-SA 4.0. Source
  2. Code: the authors state they will release code, configurations and scripts to rebuild the calibration documents. No link is given in the paper.
  3. Main results (Table 2), 10% of tokens kept, average accuracy full input vs SkimPrefill: Qwen3.5-9B 60.5 vs 62.7 (3.7x faster to first token); Gemma-4-12B 35.0 vs 45.4 (11.6x); Qwen3-VL-32B 59.6 vs 62.4 (8.2x).
  4. Time to first token on Qwen3-VL-32B (Figure 5): 19.06 s full input, of which 17.58 s prefill; 2.33 s with SkimPrefill (0.75 s preprocessing, 0.71 s selection, 0.87 s target prefill). FlexSelect 17.68 s, SGP 9.08 s, SpecPrefill 8.31 s, FastV 4.71 s.
  5. Ablation (Table 4): top single tokens using SkimPrefill's scores gave 37.9 (Qwen3.5-9B) and 15.4 (Gemma-4-12B); a random head of layer 7 gave 50.2 and 34.6.
  6. Readiness: level II, Benchmarked. Public benchmark results on MMLongBench; no released code or weights; no documented production use.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Efficiency