Trimming overlooked steps brings AI video generation to interactive speed

An Apple and Cornell team found that most of the delay in fast video generators sits outside the generation step itself. Replacing those parts with small networks brings response times under 50 milliseconds on top-end GPUs, in a preprint that has not been peer reviewed.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
A still life shows two rows of small wooden blocks threaded on string, one ending with a heavy grey stone, the second with a tiny blue glass bead.AI illustration
Illustration: AI generated

What the study shows

The study looks at autoregressive video models. These make a video piece by piece, so each new frame can respond to a new input, such as a steering command. The authors argue that frames per second is the wrong measure for this. A model can produce many frames per second yet still make a person wait a second for any reaction. They measure instead the delay from an input to the first frame that shows its effect. Their target is 50 milliseconds, a threshold taken from studies of how people perceive delay with touch and mouse input.

They first timed the fastest existing model of this kind, Causal Forcing++, on one Nvidia B200 GPU. It took 272 milliseconds per step at 480p resolution. Only about 30% of that went on generation. More than half went on the decoder, which turns the model's compressed internal output into pixels. Another big share went on an extra full pass of the model that tidies its memory of past frames, known as the KV cache.

The authors report three changes. A small network replaces the decoder, cutting it from 147 milliseconds to about 1. A second small network repairs the frame memory, taking about 2 milliseconds instead of a full model pass. Systems tuning then makes each remaining model pass about two and a half times faster on one GPU.

The 1.3 billion parameter version, FastFrames-S, responds in 36 milliseconds on one B200 GPU, about 7.6 times faster than the baseline. The 14 billion parameter version, FastFrames-L, takes 45 milliseconds but needs eight B200 GPUs. On VBench, a public automated test of video quality, both score about the same as or slightly above their closest rivals. The authors also applied the methods to two existing interactive models, one steered by camera moves and one by physical actions. Both fell from over 800 milliseconds to about 50, using eight GPUs. The authors say code and model weights are on their project page.

Our reading

In our reading, the most useful finding is the profiling itself. Teams have spent years cutting the number of generation steps. This paper shows that, for the fastest models, the generation step is no longer the main cost. The decoder and the memory refresh were treated as fixed plumbing. Measured honestly, they were the bottleneck.

The quality claim is weaker than the speed claim. On VBench, FastFrames-S beats its baseline by about a tenth of a point, and FastFrames-L leads its rival by two hundredths. The paper gives no measure of spread and no user study. We read this as "no visible loss on this benchmark", not as a quality gain. The speed numbers are clearer, because the authors break each step down and measure every model the same way.

On our readiness scale, we place this at level III, open source available. Public benchmark results are reported, and the authors say code and weights are released. We could not confirm the licence or the exact contents from the paper. There is no documented production use by a named organisation.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetWorking models, a method description and training details are given in the paper.
II  BenchmarkedMetPublic VBench results (Table 1) compare against about a dozen named baselines, with latency measured on the same B200 hardware. The numbers are self-run by the authors, not independently verified.
III  Open sourceMetThe abstract says the project page offers code and model weights: https://apple-aiml-research.github.io/ml-fastframes/. We could not check the licence or the actual contents from the paper alone.
IV  In productionNot yetNo named organisation is documented as using it in production. The only applications shown are demos built on LingBot and RealWonder.

What would move it up
Documented use in a shipped product or by a named outside organisation would give level 4. It would also help to have independent latency and quality checks (user studies, metrics beyond VBench), results on more widely available GPUs, and interactive latency at 720p or higher on one GPU.

The study

Gene Chou, Guandao Yang, Fujun Luan et al.. FastFrames: Video Generation at Interactive Latency. arXiv, 17 November 2026. Read the paper

Preprint, not peer reviewed · Licence arXiv (no reuse)

Notes

  1. Chou, Yang, Luan, Mescheder, Dong, Richter and Koltun. FastFrames: Video Generation at Interactive Latency. arXiv preprint, 17 November 2026. Not peer reviewed. Source
  2. Project page, which the authors say hosts videos, code and model weights. Licence not checked. Source
  3. Baseline profile (Causal Forcing++, 1.3B, one B200, 480p): KV cache write 41.6 ms, two-step sampling 83.2 ms, VAE decode 147 ms, total 271.8 ms. FastFrames-S: 2 ms, 32.8 ms and 1 ms, total 35.8 ms.
  4. VBench total scores (Table 1): FastFrames-S 84.23, Causal Forcing++ 84.11, FastFrames-L 84.44, Krea-Realtime 84.42.
  5. Latency on 8 B200 GPUs: FastFrames-S 14 ms, FastFrames-L 45 ms. FastFrames-L on one GPU: 158 ms. At 720p: FastFrames-S 127 ms on one GPU, FastFrames-L 116 ms on eight.
  6. Applications (Figure 5): LingBot 1.3B 842 ms to 52 ms; RealWonder 1,141 ms to 51 ms; LingBot 14B 1,373 ms to 233 ms; all optimised figures on 8 GPUs.
  7. Readiness: level III, open source available, based on the authors' statement that code and weights are released. Not yet level IV: no documented production use by a named organisation.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Efficiency