Training data for robotics
Egocentric demonstrations of real hand-object work, captured to a brief and annotated for the manipulation policy you are training.
How a demonstration becomes a delivery
-
Capture
A Capture Partner records against the brief.
The consent receipt's SHA-256 is written into a hash-chained ledger.
-
Redact
Faces the detector finds are blurred before any human review.
-
Segment
The Recording is cut into usable Clips in one pass.
Each Clip id carries the first eight characters of its SHA-256.
-
Annotate
A model estimates 3D hand pose and confidence, both derived.
-
Deliver
The annotated data ships in the container the brief names.
The package manifest lists each file's SHA-256.
Recompute them with
shasum -a 256.
What a manipulation policy learns from
A manipulation policy learns from demonstrations. Three published results show how much depends on them:
- The ACT paper reports that its robot learned six difficult real-world tasks, “such as opening a translucent condiment cup and slotting a battery with 80-90% success, with only 10 minutes worth of demonstrations”, collected by teleoperation (Zhao et al., arXiv:2304.13705).
- The robomimic study lists dependence on the quality of the demonstrations among its main lessons for learning from offline human data (Mandlekar et al., arXiv:2108.03298).
- The DROID authors write that even the most general manipulation policies are mostly trained on data from a small number of environments, with limited scene and task diversity (Khazatsky et al., arXiv:2403.12945).
Every figure above is the paper's own. A brief settles the task, the environment, the device and the annotation tier before anything is recorded.
Training a vision-language-action model instead? The VLA training data guide sets out what published runs trained on.
Delivery formats
Every export is read back off disk by a separate checker before it ships. MCAP and WebDataset are not offered: MCAP is a logging container from the ROS 2 and Foxglove lineage, and WebDataset sharding belongs to the training run. The delivery page sets out each container and its check.
| Format | Status | What it is |
|---|---|---|
| LeRobot v2.1 | Standard | Parquet, MP4 and JSONL in the LeRobot dataset layout. It holds no robot actions: retargeting a human hand into your robot's action space is your modelling step |
| HDF5 | Standard | One .hdf5 file beside one .mp4, the paired layout EgoDex publishes |
| FH-Ego v1 JSON | Standard | Our open annotation schema, as MP4 plus JSON, with every field documented |
| Zarr v2 | On request | The chunked-array layout UMI and Diffusion Policy train from |
| TFRecord, RLDS-style | On request | An RLDS-style step schema; no tfds.load dataset builder is included |
| MCAP | Not offered | ROS 2 logging and replay, not policy training |
| WebDataset | Not offered | A sharding layout that belongs to the training run |
Scope a manipulation brief
Start from the task, not the hours. Name the task, the environment and the policy, and we scope a Capture Brief around them.