Smaller, faster and cheaper AI: research on compression, quantisation, distillation and efficient inference, and what it means for models in production.
Kumo Tabular predicts on spreadsheet-style data without training on it. It ranks first on eight public tests and is fast, but several leads sit within statistical noise.
An Apple and Cornell team found that most of the delay in fast video generators sits outside the generation step itself. Replacing those parts with small networks brings response times under 50 milliseconds on top-end GPUs, in a preprint that has not been peer reviewed.
A 7B model running on one graphics card answered about half of the questions sent to a costly AI agent. In the authors' tests this cut list-price spending by a fifth to a third, with no measurable drop in accuracy. The work is a preprint and has not been peer reviewed.
A computer model of 4,317 US sites finds that solar panels and batteries wired for direct current win in most places once lost computing time is priced in. The study is a preprint and has not been peer reviewed.
A single-author preprint argues that popular losses for training draft models measure the wrong thing, and proposes an exact alternative. The gains on one 35B model are consistent but small.