Smaller, faster and cheaper AI: research on compression, quantisation, distillation and efficient inference, and what it means for models in production.
A tiny vision-language model skims long image-and-text inputs and hands a larger model only the pages, images and paragraphs that matter. On MMLongBench, three larger models started answering several times sooner, with average accuracy held or slightly raised.
An MIT and NVIDIA preprint lets a coding agent search for the best way to compress each reasoning model. It gains about one to two accuracy points over the strongest public recipes for an 8-hour search on 8 GPUs, and slightly beats NVIDIA's own compressed release of a 320B model.
A preprint adapts off-the-shelf speech recognition models to turn recorded brain activity into text on an iPhone. It uses a small fraction of the memory that current research systems need. It has not yet been tried with a live user.
A preprint from Together AI describes a loop where AI agents rewrite and test the code that runs large models, again and again as traffic changes. The authors report up to 2.6 times higher throughput than the best public entries on a standard benchmark.