VLA training data guide
What published vision-language-action models were trained on, which container a delivery can arrive in, and how both shape a brief.
Where annotation and export sit
-
Capture
A Capture Partner records against the brief.
The consent receipt's SHA-256 is written into a hash-chained ledger.
-
Redact
Faces the detector finds are blurred before any human review.
-
Segment
The Recording is cut into usable Clips in one pass.
Each Clip id carries the first eight characters of its SHA-256.
-
Annotate
A model estimates 3D hand pose and confidence, both derived.
-
Deliver
The annotated data ships in the container the brief names.
The package manifest lists each file's SHA-256.
Recompute them with
shasum -a 256.
What published VLA runs trained on
RT-2 named the model class: a vision-language model co-fine-tuned on robot trajectories and web vision-language tasks, with actions written as text tokens (Brohan et al., arXiv:2307.15818). The runs and datasets below show the range since. Every figure belongs to the paper linked beside it.
| Run or dataset | Trained on, in the paper's terms | Source |
|---|---|---|
| OpenVLA | 970k real-world robot demonstrations | Kim et al., arXiv:2406.09246 |
| Open X-Embodiment | Data from 22 robots across 21 institutions, covering 527 skills | Open X-Embodiment Collaboration, arXiv:2310.08864 |
| DROID | 76k demonstration trajectories, or 350 hours of interaction data, in 564 scenes | Khazatsky et al., arXiv:2403.12945 |
| π0 | A large, diverse dataset from single-arm, dual-arm and mobile manipulator robots | Black et al., arXiv:2410.24164 |
| GR00T N1 | A mix of real-robot trajectories, human videos and synthetic data | NVIDIA, arXiv:2503.14734 |
These are what published runs used, not a quota for yours. A brief sizes capture to your task and to the gap in your data. Where human egocentric video fits in such a mix is argued, source by source, on Why egocentric.
Containers a delivery can arrive in
Standard: LeRobot v2.1, HDF5 in the EgoDex paired layout, and FH-Ego v1 JSON. On request: Zarr v2, and TFRecord with an RLDS-style step schema. Not offered: MCAP, a logging container, and WebDataset, a sharding layout for the training run.
A VLA learns actions, and the LeRobot export carries observations only: mapping a human hand to your robot's action space stays your modelling step. The TFRecord export does not load through tfds.load without a dataset builder. The delivery page covers each container and how it is checked.
Scope a VLA brief
Tell us the model, the tasks and the container your training run reads, and we scope capture against the gap in your data.