Smaller, faster and cheaper AI: research on compression, quantisation, distillation and efficient inference, and what it means for models in production.
A tiny vision-language model skims long image-and-text inputs and hands a larger model only the pages, images and paragraphs that matter. On MMLongBench, three larger models started answering several times sooner, with average accuracy held or slightly raised.
An MIT and NVIDIA preprint lets a coding agent search for the best way to compress each reasoning model. It gains about one to two accuracy points over the strongest public recipes for an 8-hour search on 8 GPUs, and slightly beats NVIDIA's own compressed release of a 320B model.
Kumo Tabular predicts on spreadsheet-style data without training on it. It ranks first on eight public tests and is fast, but several leads sit within statistical noise.
An Apple and Cornell team found that most of the delay in fast video generators sits outside the generation step itself. Replacing those parts with small networks brings response times under 50 milliseconds on top-end GPUs, in a preprint that has not been peer reviewed.
A 7B model running on one graphics card answered about half of the questions sent to a costly AI agent. In the authors' tests this cut list-price spending by a fifth to a third, with no measurable drop in accuracy. The work is a preprint and has not been peer reviewed.
A preprint adapts off-the-shelf speech recognition models to turn recorded brain activity into text on an iPhone. It uses a small fraction of the memory that current research systems need. It has not yet been tried with a live user.
A preprint from Together AI describes a loop where AI agents rewrite and test the code that runs large models, again and again as traffic changes. The authors report up to 2.6 times higher throughput than the best public entries on a standard benchmark.
A computer model of 4,317 US sites finds that solar panels and batteries wired for direct current win in most places once lost computing time is priced in. The study is a preprint and has not been peer reviewed.
A single-author preprint argues that popular losses for training draft models measure the wrong thing, and proposes an exact alternative. The gains on one 35B model are consistent but small.