Kyutai streams talking-face video from live speech on one H100 GPU

Researchers at Kyutai shrank a slow talking-head video model into one that keeps pace with speech. The smaller version runs faster than real time on a single GPU. The work is a preprint and has not been peer reviewed.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
An artistic studio still life featuring a paper strip with oval cut-outs unspooling alongside a wavy string.AI illustration
Illustration: AI generated

What the study shows

Talking-head models take a reference photo and a speech recording and produce video of that person speaking. The best open ones, such as TalkVerse and Wan-S2V, are slow. They generate several seconds of video at once, over dozens of refinement passes, and need to hear audio ahead of time. That rules them out for live conversation.

The authors, mostly from the Paris lab Kyutai, built a family of models they call AURA. It produces video in short chunks of about two thirds of a second. Each chunk needs only the audio for that same stretch, not future speech. It uses four refinement passes per chunk instead of 50, and keeps a small fixed memory of recent chunks plus the very first one, so memory use does not grow as the video gets longer.

They made it by distillation, where a fast 'student' model is trained to copy a slow 'teacher'. The larger AURA 5B (5 billion parameters, the adjustable weights in a model) learned from TalkVerse. The smaller AURA 1.3B learned from a teacher the authors trained themselves. Training used a set of 30,000 synthetic video clips the teacher generated from computer-made speech.

On one Nvidia H100 GPU, the authors report that AURA 5B makes 720p video at 9.7 frames per second, about 17 times faster than TalkVerse. AURA 1.3B makes lower-resolution video at about 21 frames per second, above its 16 frames per second playback rate, so it runs in real time. Both were faster than every rival tested, including Live Avatar, a 14-billion-parameter streaming model, which managed 1.9 frames per second.

On quality, results are mixed. AURA 5B beat TalkVerse on a standard video realism score, and kept the person's identity closest to the photo of all models tested. The older Hallo3 model scored better on the realism scores. In a human study with 27 raters and 400 side-by-side comparisons, both AURA models were preferred over Hallo3 on all four measures. AURA 5B was on par with TalkVerse on quality, lip-sync and naturalness. AURA 1.3B ranked below both on those three.

Our reading

In our reading, the main result is the smaller model running faster than playback on one card. Earlier streaming talking-head systems of this kind used models of 14 to 18 billion parameters and, the authors say, needed four or five GPUs. Going down to one card changes who can afford to run a live face. A customer-service voice bot could show a talking agent next to its replies, without a cluster behind each conversation.

The quality picture needs care. Standard video scores favour Hallo3, yet human raters preferred AURA, and the authors point to hand and mouth artefacts the scores miss. That is a useful warning about automatic metrics in general. But it also means AURA's quality case rests heavily on a small human study the authors ran themselves, on their own test sets.

On our readiness scale we place this at level III, open source available, provisionally. The paper says weights for both models, the dataset and the small teacher are released, with a pointer to the project page. We have not yet confirmed the download or the licence terms. There is no documented use by an outside organisation, so level IV is not met.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetMethod described in full, with training recipe, ablations and failure analysis in the supplementary material.
II  BenchmarkedMetCompared with Hallo3, Wan-S2V, TalkVerse and Live Avatar on two test sets using FID, FVD, JEDi, identity match, lip-sync and warping error, plus a human preference study. Caveat: the test sets and comparison were set up by the authors, and no independent standard streaming benchmark was used.
III  Open sourceMetThe paper states that weights for both AURA models, the 30K-clip dataset with teacher trajectories, and the 1.3B teacher are released. The paper gives no repository link in the text I could read, only the project page kyutai.org/aura. I could not verify the release or its licence from the paper alone.
IV  In productionNot yetNo named organisation reports using AURA in production. The only deployment is the authors' own live demo with Kyutai's Unmute voice assistant.

What would move it up
To reach level 4, a named organisation would need to run AURA in a real product with live users, and report latency, cost per stream and failure rates. Independent benchmarks run by third parties, and a published end-to-end latency figure, would also strengthen the case. Confirmation of the weights, code and licence terms would settle level 3.

The study

Nathaniel Cohen, Nicolas Dufour, Amélie Royer et al.. AURA: AUdio-dRiven streaming Avatar. arXiv, 19 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY NC SA 4.0

Notes

  1. Cohen, Dufour, Royer, Newson and Pérez, 'AURA: AUdio-dRiven streaming Avatar', arXiv preprint 2611.26540, November 2026. Licence: CC BY-NC-SA 4.0. Source
  2. Project page with videos, live demo and, per the authors, released weights and dataset. Source
  3. Exact speeds on one H100, end to end for a 10 s clip: AURA 5B 9.7 fps, TalkVerse 0.58 fps, AURA 1.3B 20.7 fps (17.3 fps at 768x512), Live Avatar 1.9 fps, Hallo3 0.22 fps, Wan-S2V 0.08 fps. AURA 1.3B teacher: 0.90 fps on one 4.8 s window.
  4. Exact quality figures on the Hallo3 test set: FVD (lower is better) AURA 5B 195.5, TalkVerse 210.8, Hallo3 183.4, AURA 1.3B 177.1; identity match (CSIM) AURA 5B 0.79, AURA 1.3B 0.84. On CelebV-HQ test: FVD AURA 5B 208.1, TalkVerse 255.9, Hallo3 202.5.
  5. One-minute test: AURA 1.3B FVD 179 to 186 across six 10 s segments; AURA 5B FVD rose from 182 to between 195 and 206, and identity match fell from 0.76 to 0.69.
  6. TalkVerse, the teacher model for AURA 5B: Wang et al., 2025. Source
  7. Live Avatar, the closest streaming competitor: Huang et al., 2026. Source

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Vision & multimodal