# Small Qwen3.5 model picks the evidence, cutting long-document wait up to 11.6x

A tiny vision-language model skims long image-and-text inputs and hands a larger model only the pages, images and paragraphs that matter. On MMLongBench, three larger models started answering several times sooner, with average accuracy held or slightly raised.

- Category: Efficiency
- Readiness level: II (Benchmarked) on Science Underway's four-step scale, as proposed
- Published: 2026-11-19
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/small-qwen3-5-model-picks-the-evidence-cutting-long-document-wait-up-to-11-6x

## The study

Let a Small Model Read First: Faster Prefill for Long Interleaved Image-Text Inputs (preprint, not peer reviewed). arXiv 2611.24431. https://arxiv.org/abs/2611.24431

## At a glance

- What it shows: A very small AI model can skim a long mix of pictures and text, pick the few images and paragraphs that answer a question, and pass only those to a bigger model. On a public test, three bigger models started answering about 4 to 12 times sooner, and their average scores held up or improved.
- What it does not show: It does not show that every task stays as good. Summaries and some questions about long documents got worse. All tests used one benchmark and one hardware setup, and the code is promised but not yet released.
- Why it matters: Questions about long documents or large photo collections can leave users waiting tens of seconds before the first word appears. This approach cuts that wait without retraining the big model or changing how it is served.

## What the study shows

Vision-language models answer questions about images and text together. When the input is long, such as a report of many pages or a few hundred photos, they are slow to start. Most of that wait is the prefill: the step where the model reads the whole input before writing anything. The authors report that on Qwen3-VL-32B, an input of about 105,000 tokens (chunks of text or image) waited about 19 seconds for the first output word. Almost all of it was prefill.

The method, which the authors call SkimPrefill, adds a small selector model in front. The selector is Qwen3.5-0.8B, a hybrid model in which most layers take time that grows only in step with input length. It reads the whole input with images at half resolution, but stops after 8 of its 24 layers. One attention head there, a part of the model that weighs how strongly the question relates to each piece of the input, scores every image and every paragraph. The top-scoring units, about 10% of the tokens, go to the larger target model as an ordinary, shorter prompt. The head was chosen on 50 synthetic documents that share nothing with the test data. No training was needed.

The authors tested on MMLongBench at its longest setting, about 128,000 tokens: 1,750 questions from 19 datasets across five task types. The targets were Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B. They compared against four published pruning methods that need no training: FastV, FlexSelect, SGP and SpecPrefill. All kept the same 10% of tokens.

The authors report that time to first token fell by 3.7 times on Qwen3.5-9B, 11.6 times on Gemma-4-12B and 8.2 times on Qwen3-VL-32B. Average accuracy rose slightly on the two Qwen targets and by about 10 points on Gemma-4-12B. SkimPrefill had the best average of the pruning methods on all three targets. Most rivals gave little or no speed gain on two of the targets, and the fastest rival, FastV, lost more than 20 points.

The gains were uneven. Needle-in-a-haystack questions, where the answer sits in a few images or sentences, rose by 11 to 26 points. Summarisation, which needs the whole document, fell by 3 to 4 points. On one long-document dataset, LongDocURL, the selector kept no answer page at all for 18 of 100 questions.

The targets, which reason step by step before answering, also wrote shorter reasoning with SkimPrefill. On Qwen3.5-9B, average output fell from about 6,400 to 3,500 tokens. The full time to a finished answer was 2.2 to 3.8 times shorter. This is a preprint and has not been peer reviewed. On our readiness scale it sits at level II, Benchmarked: public benchmark results exist, but no code has been released yet.

## Our reading

In our reading, the most useful finding is about what to keep, not how to score it. Earlier methods kept the most important single tokens. On long mixed inputs those tokens end up scattered: a few patches of most images and fragments of most paragraphs. The authors show this breaks the evidence apart. When they kept the top single tokens using SkimPrefill's own scores, accuracy collapsed on both targets tested. Keeping whole images and paragraphs with the same scores did not. Whole units also raised the accuracy of two rival scoring methods.

The headline speed figures need context. The largest gain, on Gemma-4-12B, partly reflects a weak starting point. With the full input, Gemma often hit its 32,000-token reasoning limit and scored close to zero on one task type, many-shot in-context learning. The authors report that Gemma scored higher there the less input it read. So the 10-point gain on Gemma says as much about Gemma struggling with very long inputs as about the selector. The Qwen results, a 2 to 3 point gain at 3.7 to 8.2 times faster, are the more representative picture.

A concrete case: an assistant answering questions over a bundle of scanned product manuals, or a few hundred site-inspection photos. If the answer sits in a handful of pages or pictures, this approach could cut a 20-second pause to a few seconds. If the user asks for a summary of the whole bundle, it would likely do worse than reading everything.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/small-qwen3-5-model-picks-the-evidence-cutting-long-document-wait-up-to-11-6x

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
