# Robots learn CALVIN tasks from human demonstrations, but still need robot play

A Freiburg-led team trained robot arm skills from people's demonstrations, which record no robot commands, and then improved them inside a learned simulator. Robot recordings are still needed to translate those skills into motor commands.

- Category: Robotics
- Readiness level: III (Open source) on Science Underway's four-step scale, as proposed
- Published: 2026-11-19
- Editor: Stefan Perdal
- URL: https://scienceunderway.com/articles/robots-learn-calvin-tasks-from-human-demonstrations-but-still-need-robot-play

## The study

DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play (preprint, not peer reviewed). arXiv 2611.24836. https://arxiv.org/abs/2611.24836

## At a glance

- What it shows: A robot arm learned simple tasks, such as opening a drawer, mostly by watching a person do them. In a standard robot simulation, this beat methods trained on five robot demonstrations per task.
- What it does not show: It does not show robots learning from ordinary online video, or doing without robot data. A few hours of the robot moving freely were still needed, and the real-robot test was small.
- Why it matters: Recording a robot doing a task is slow and costly, while a person can show it in seconds. Shifting most of the teaching to people could cut the cost of training robots for new jobs.

## What the study shows

The work tackles a basic bottleneck in robot learning. Teaching a robot usually needs demonstrations that include its exact motor commands. These are recorded by a person steering the robot remotely, which costs an hour of human and robot time per hour of data. A person doing the task by hand is much faster, but that footage records no robot commands.

The method, called DiWALI by its authors, builds a world model: a learned simulator that predicts what the camera will see next. Instead of feeding it robot commands, it infers a compact 'latent action' from two consecutive frames. This describes the change between frames, whether a hand or a gripper caused it. So footage of people and of the robot can train the same model. A small separate decoder, trained only on robot recordings, turns these latent actions into real robot commands.

The robot's skill is a diffusion policy, a model that generates actions by gradually refining random noise. It is trained to copy the latent actions seen in human demonstrations. It is then improved with reinforcement learning, where a model learns from rewards, entirely inside the world model. No real robot time is used for this step.

The authors tested this on CALVIN, a public simulated tabletop benchmark, over 11 tasks. Here the 'human' is a rendered arm placed in the scene, not real people. With 50 human demonstrations and no robot demonstrations per task, the policy succeeded about 60% of the time. The best comparison method, given five robot demonstrations per task, reached about 35%. After improvement in the learned simulator, the best DiWALI setting reached about 76%.

On a real Franka Emika Panda arm, the team tested three skills: opening a drawer, sliding a door and pressing a button for a light. Policies learned mostly from human demonstrations succeeded in 53% of attempts. After training in the learned simulator, this rose to 70%, with no extra real-world practice. The paper is a preprint and has not been peer reviewed.

## Our reading

In our reading, the most useful result is not the headline success rate but where the method's limit lies. Human footage can replace most robot footage, but not all. When robot recordings fell to 1% of the six hours of free play, a few minutes, success collapsed to about 7%. At a quarter robot, three quarters human, it was nearly as good as the default half-and-half mix. Robot data stops being the bulk of the work but stays the anchor.

The second point is how the gap with a similar method, CLAM, is explained. Using the same data, the authors' version of CLAM scored about 8% against about 60%. Their step-by-step tests trace most of this to one choice: letting the decoder see the robot's current situation, not only the intended change. That is a plain, reusable design lesson. It comes with a caveat: CLAM was rebuilt by the authors inside their own system, not run from its original code.

A practical example: a team setting up a robot to open a specific cabinet could record its own staff doing the task 50 times by hand. Today it would need to steer the robot remotely through each demonstration. The method suggests that the hand-made recordings could carry most of the teaching load, once the robot has some free-play recordings in that same workspace.

The rest of the article (what it means, limits, what to watch) is free for members: https://scienceunderway.com/articles/robots-learn-calvin-tasks-from-human-demonstrations-but-still-need-robot-play

## How this was made

An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.

---
Source: Science Underway, https://scienceunderway.com/. Please link to the article when you cite it.
