Physical AI training data
Egocentric video of people doing real physical work, captured to a brief and annotated for policy and world-model training.
From a brief to a delivered file
-
Capture
A Capture Partner records against the brief.
The consent receipt's SHA-256 is written into a hash-chained ledger.
-
Redact
Faces the detector finds are blurred before any human review.
-
Segment
The Recording is cut into usable Clips in one pass.
Each Clip id carries the first eight characters of its SHA-256.
-
Annotate
A model estimates 3D hand pose and confidence, both derived.
-
Deliver
The annotated data ships in the container the brief names.
The package manifest lists each file's SHA-256.
Recompute them with
shasum -a 256.
What physical AI is trained on
In NVIDIA's words, physical AI lets autonomous systems such as robots and self-driving cars “perceive, understand, reason, and perform or orchestrate complex actions in the physical world” (NVIDIA glossary). Its Cosmos paper splits what such a system needs in two: a policy model for the system itself, and a world model for its surroundings (arXiv:2501.03575).
GR00T N1, NVIDIA's open foundation model for humanoid robots, was trained on a mix of real-robot trajectories, human videos and synthetic data (arXiv:2503.14734). Human video is the part of that mix a brief commissions: egocentric video of hands doing the task, in the environment you name.
What the annotation layer carries
An annotated Clip is its video plus an FH-Ego v1 annotation file, and together they carry the signals below. The last column marks the recorded footage as measured, and everything software produced as derived.
| Signal | What it holds | Provenance |
|---|---|---|
| Video | The Clip itself; faces the detector finds are blurred before anyone reviews it | Measured |
| Hand keypoints | 21 points per detected hand, from the MediaPipe hand landmark model, on sampled frames | Derived |
| Wrist position and velocity | Where each detected hand's wrist sits in the image, with a depth value, and how fast it is moving. The values have no physical unit | Derived |
| Per-hand confidence | The hand detector's score for whether each detected hand is a left or a right hand | Derived |
| Action windows | Windows across the Clip, each with a verb-noun code | Derived |
| Captions | One text description per action window, and the file names the captioner that wrote them | Derived |
FH-Ego v1 documents every field, with one real Clip's values beside it.
Scope a physical AI brief
Tell us what your model has to learn and where that work happens. We scope a Capture Brief for the footage and the annotation tier it needs.