Robots learn CALVIN tasks from human demonstrations, but still need robot play

A Freiburg-led team trained robot arm skills from people's demonstrations, which record no robot commands, and then improved them inside a learned simulator. Robot recordings are still needed to translate those skills into motor commands.

6 min readIIIOpen sourcePreprint, not peer reviewed
Save
A still life of a wooden toy cabinet and tracing paper on warm paper, connected by a blue thread.AI illustration
Illustration: AI generated

What the study shows

The work tackles a basic bottleneck in robot learning. Teaching a robot usually needs demonstrations that include its exact motor commands. These are recorded by a person steering the robot remotely, which costs an hour of human and robot time per hour of data. A person doing the task by hand is much faster, but that footage records no robot commands.

The method, called DiWALI by its authors, builds a world model: a learned simulator that predicts what the camera will see next. Instead of feeding it robot commands, it infers a compact 'latent action' from two consecutive frames. This describes the change between frames, whether a hand or a gripper caused it. So footage of people and of the robot can train the same model. A small separate decoder, trained only on robot recordings, turns these latent actions into real robot commands.

The robot's skill is a diffusion policy, a model that generates actions by gradually refining random noise. It is trained to copy the latent actions seen in human demonstrations. It is then improved with reinforcement learning, where a model learns from rewards, entirely inside the world model. No real robot time is used for this step.

The authors tested this on CALVIN, a public simulated tabletop benchmark, over 11 tasks. Here the 'human' is a rendered arm placed in the scene, not real people. With 50 human demonstrations and no robot demonstrations per task, the policy succeeded about 60% of the time. The best comparison method, given five robot demonstrations per task, reached about 35%. After improvement in the learned simulator, the best DiWALI setting reached about 76%.

On a real Franka Emika Panda arm, the team tested three skills: opening a drawer, sliding a door and pressing a button for a light. Policies learned mostly from human demonstrations succeeded in 53% of attempts. After training in the learned simulator, this rose to 70%, with no extra real-world practice. The paper is a preprint and has not been peer reviewed.

Our reading

In our reading, the most useful result is not the headline success rate but where the method's limit lies. Human footage can replace most robot footage, but not all. When robot recordings fell to 1% of the six hours of free play, a few minutes, success collapsed to about 7%. At a quarter robot, three quarters human, it was nearly as good as the default half-and-half mix. Robot data stops being the bulk of the work but stays the anchor.

The second point is how the gap with a similar method, CLAM, is explained. Using the same data, the authors' version of CLAM scored about 8% against about 60%. Their step-by-step tests trace most of this to one choice: letting the decoder see the robot's current situation, not only the intended change. That is a plain, reusable design lesson. It comes with a caveat: CLAM was rebuilt by the authors inside their own system, not run from its original code.

A practical example: a team setting up a robot to open a specific cabinet could record its own staff doing the task 50 times by hand. Today it would need to steer the robot remotely through each demonstration. The method suggests that the hand-made recordings could carry most of the teaching load, once the robot has some free-play recordings in that same workspace.

Free membership

Read the rest of this story

Membership is free. You get every article in full and the Issue every Sunday.

We send a sign-in link, no password needed. Already a member? Sign in

Readiness assessment

III
Open source
LevelStatusEvidence
I  Lab onlyMetWorking method with simulated and real-robot experiments, described in full including hyperparameters and pseudocode.
II  BenchmarkedMetEvaluated on the public CALVIN benchmark (environment D, 11 tasks, three seeds) against Diffusion Policy, DiWA and a re-implemented CLAM. This is a partial benchmark setting: the authors split the data and built their own human-play set (CALVIN-H), so numbers are not directly comparable with other CALVIN leaderboards.
III  Open sourceMetThe abstract and contributions say code, dataset and videos are available at https://diwali.cs.uni-freiburg.de. We could not check the page or the licence terms from the paper alone.
IV  In productionNot yetNo documented use by a named organisation outside the authors' own lab. The only real-world test is three simple skills on one arm in the authors' setup.

What would move it up
To reach level 4, DiWALI would need documented use outside the authors' lab, for example a named company or other group running it on its own robot and tasks. It would also help to test with real human video rather than a rendered arm in simulation, include a real-world baseline, and cover longer, multi-step tasks and scenes with changing conditions. Independent replication on CALVIN or another public benchmark would strengthen level 2.

The study

Akshay L Chandra, Iman Nematollahi, Lingyue Zhao et al.. DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play. arXiv, 18 November 2026. Read the paper

Preprint, not peer reviewed · Licence CC BY NC SA 4.0

Notes

  1. Chandra, Nematollahi, Zhao, Kulikova, Schoellig, Despinoy and Valada, 'DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play', arXiv preprint, 18 November 2026, licence CC BY NC SA 4.0. Source
  2. Project page with videos, dataset and code, as stated by the authors. Contents and licence not checked by us. Source
  3. CALVIN results, Table I: 59.6% average success over 11 tasks with 50 human and 0 robot demonstrations per task; 61.6% with 5 robot added; Diffusion Policy (ViT) 35.2% and DiWA 23.6% with 5 robot demonstrations, falling to 11.1% and 4.2% with one. CLAM-X with the same data: 8.2%.
  4. Ablation, Table II: decoder trained separately 18.7%, decoder conditioned on latent state 61.6%. Robot share of play data: 1% gives 7.4%, 10% gives 36.2%, 25% gives 58.0%, 50% gives 61.6%. Optical flow 61.3%, point flow 63.5%.
  5. Fine-tuning in the world model, Table III: DiWALI with 5 robot demonstrations 48.1% to 67.4%; with 50 human added 61.6% to 75.8%. DiWA (50% robot play, 5 demos) 23.6% to 50.4%; (50%, 50) 47.0% to 69.5%; (100%, 50) 54.7% to 80.4%.
  6. Real robot: Franka Emika Panda, two hours of robot play and two hours of human play, 50 human and 5 robot demonstrations per skill, 20 rollouts per skill over three seeds; 53% before and 70% after fine-tuning.
  7. CALVIN benchmark: Mees et al., IEEE Robotics and Automation Letters, 2022.

Edited and signed by

Stefan PerdalEditor and publisher86 stories edited

Stefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.

How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

More on Robotics