Turning 19 human grasp recordings into 25,000 robot training demonstrations
A preprint describes DexSeed, a method that expands a few recordings of human hands into simulated robot training data. A robot trained only on that data picked and placed real objects, with mixed success.

AI illustrationWhat the study shows
Robot hands with many joints need large amounts of example data to learn. The usual source is teleoperation, where a person steers the robot remotely while it records. For a hand with 20 finger joints this is hard and slow. The authors ask whether existing recordings of human hands can be expanded into usable robot training data instead. The paper is a preprint and has not been peer reviewed.
The method, called DexSeed, starts from GRAB, a public dataset of people grasping objects. The authors grouped the recordings by grasp type and chose 8 types by hand, such as a top-down whole-hand grasp or threading fingers through a handle. Each type gets one text description, written by GPT. They then converted 19 human recordings to a robot hand, a step called retargeting. Each converted motion was reused with differently shaped objects, start positions and drop-off points.
Next, the motions were run in a physics simulator. A finger controller was trained by reinforcement learning, where a model learns from rewards, while the arm followed its planned path. Only rollouts that succeeded were kept. Camera images were then rendered from a 3D reconstruction of the real lab bench. The result was 25,270 demonstrations, each pairing images, robot actions and a grasp description.
The authors report gains over the methods they compared against at each stage. Their retargeting kept fingertip contacts and angles closer to the human original than three alternatives, though not with less hand-object overlap than the best one. With the arm in the loop, their finger controller succeeded in about 96% of simulated rollouts. The two comparison methods managed about a quarter to a third.
They then fine-tuned a vision-language-action model, which maps camera images and a text instruction to robot movements. In simulation it succeeded in over 90% of pick-and-place trials, including with unseen object shapes, positions and colours. Models trained on data from the comparison methods scored roughly 26% to 51% in those unseen settings. Removing the grasp descriptions lowered success on seen conditions from 97.5% to 80%.
On a real Franka arm with a Wuji hand, the model trained only on simulated data succeeded in 20% to 80% of trials, depending on grasp type. Handle threading was hardest. Two-finger hooking was easiest. In a test on one apple, the robot used the grasp it was told to use in all 10 trials for each of two instructions.
Our reading
In our reading, the most useful result is not the headline accuracy in simulation. It is that a text instruction reliably switched between two different grasps on the same object. That suggests the grasp descriptions do real work, not just decoration. In practice, a kitchen or warehouse robot could be told to hold a mug by the handle or wrap it, without new demonstrations for each choice.
The comparison with other methods needs care. The authors trained their finger controller with the arm moving. The comparison methods were trained with a free-floating hand, and they did well in that setting. Under arm tracking they fell behind. So the gap shows that training with the arm matters, which is the authors' point. It is not a like-for-like contest between methods.
On the readiness scale, we rate this level II, Benchmarked. The authors report results against named methods on a public dataset, but on a test suite they built themselves. The paper links only a project page and does not state a release of code, data or model weights, so level III is not met on the evidence in the paper.
Read the rest of this story
Membership is free. You get every article in full and the Issue every Sunday.
We send a sign-in link, no password needed. Already a member? Sign inReadiness assessment
| Level | Status | Evidence |
|---|---|---|
| I Lab only | Met | The method works in simulation and in a small real-robot demonstration. |
| II Benchmarked | Met | The paper reports comparisons against named baselines (Dex-Retargeting, SPIDER, Wuji official, ManipTrans, Dexplore) on the same 19 sequences and a 320-trial simulation suite. GRAB is a public dataset, but the evaluation suite is the authors' own, not a shared public benchmark. We rate this met, with that caveat. |
| III Open source | Not yet | The paper links only a project page (https://robbyant-research.github.io/DexSeed/). No code, weights or dataset release is stated in the text. A release may exist on that page, which we could not check. |
| IV In production | Not yet | No documented use by a named organisation outside the research setting. The real-robot test is a lab demonstration. |
What would move it up
Releasing the code, the 25,270-demonstration dataset or trained weights would support level 3, if the licences allow reuse. For level 4, an independent team or a named company would need to use it on real tasks. Bigger real-world tests would also help: more objects per mode, more trials and a comparison with teleoperated data.
Checked automatically every week, last on 16 November.
The study
Huaijin Pi, Zehong Shen, Yong Wang et al.. DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning. arXiv, 10 November 2026. doi:10.48550/arxiv.2611.14092
Notes
- Pi et al., DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning. arXiv preprint 2611.14092, November 2026. Licence CC BY NC SA. Source
- Project page. The paper does not state whether code, data or weights are released. Source
- Retargeting (Table II): fingertip contact error 3.326 mm against 3.521 (SPIDER), 5.273 (Wuji official) and 9.486 (Dex-Retargeting). Angular error 7.636 degrees against 16.350 to 18.560. Penetration 3.371 mm against 2.959 mm for Wuji official.
- Physics-based imitation with arm tracking (Table III): success 95.50% against 34.25% (ManipTrans) and 24.81% (Dexplore). Floating-hand success: 71.32% and 94.33%. Variant without arm in training: 28.02% with arm, 98.04% floating.
- Simulated VLA success (Table IV), seen / unseen layout / plus geometry / plus appearance: DexSeed 97.50 / 95.00 / 91.25 / 91.25. ManipTrans data 38.75 to 51.25. Dexplore data 26.25 to 40.00. Kinematic data 5.00 to 12.50. Without grasp text 80.00 / 80.00 / 76.25 / 70.00. pi0 on the same data 88.75 / 88.75 / 86.25 / 90.00.
- Data scale (Table VI): 1% of data gave 67.50 / 38.75 / 31.25 / 28.75; 10% gave 85.00 / 87.50 / 90.00 / 85.00.
- Real-world success by grasp type (Table VII), G1 to G8: 60, 40, 70, 70, 20, 50, 50, 80%. Mean of 55% is our calculation.
- Suggested figure: Figure 2, the pipeline overview. Reuse is non-commercial only under the CC BY NC SA licence.
Edited and signed by
Stefan PerdalEditor and publisher86 stories editedStefan read the study, set the readiness level and approved this story. An AI system found the paper and wrote the first draft.
How this was made
An AI model read the study and wrote this article in the structure Science Underway uses. An editor reviewed it and approved it for publication. Check the original study before you rely on the details.
How we workNo corrections


