Capture catalogue
Robot-ready capture, specified before it is collected
Every entry below is a capture specification we can scope and deliver: domain, sensor stack, annotation tier and delivery format fixed up front, so you know exactly what arrives. Read the FH-Ego v1 schema on any entry before committing to a pilot.
A target volume is what a full engagement commits to — agreed per brief before capture, rather than drawn from stock. Every entry states where it stands:
Pipeline validated the annotation pipeline has been run end-to-end on a real recording of this task family
Scoped on demand quoted per engagement
Research phase under evaluation; not open for commission
182,700 clips and 1,505 hours of egocentric capture are commissionable across 6 domains Target; these are volumes a brief commits to. Each specification below states its own availability. The totals leave out the simulated-traversal and RLHF briefs, which record no egocentric footage.
8 of 8 specifications
| Specification | Domain | Target volume | Availability | Capture device | Camera | Spatial tracks | Annotation |
|---|---|---|---|---|---|---|---|
| Precision Factory Assembly | Manufacturing | 35,000 clips / 290 hrs Target | Pipeline validated | Stereo Global-Shutter Head Rig | 2560x960 @ 90 FPS (Stereo) | 6-DoF Camera Trajectory (SLAM) & Sync IMU (400Hz) | Tier 1 · Tier 2 · Tier 3 |
| Bimanual Household Tasks | Kitchen | 42,500 clips / 350 hrs Target | Scoped on demand | Ray-Ban Meta | 1920x1080 @ 30 FPS | 3D Camera-Relative Cartesian Coordinates (X, Y, Z) | Tier 1 · Tier 2 · Tier 3 |
| Specialized Caregiving & Nursing | Healthcare | 18,200 clips / 150 hrs Target | Scoped on demand | GoPro Hero 12 | 3840x2160 @ 60 FPS | IMU Synchronized Telemetry (200Hz) | Tier 1 · Tier 2 |
| Warehouse & Logistics | Logistics | 50,000 clips / 410 hrs Target | Scoped on demand | GoPro Hero 12 | 1920x1080 @ 30 FPS | Wrist-mounted IMU sensor synchronization | Tier 1 · Tier 2 |
| Cleaning & Maintenance | Household | 22,000 clips / 180 hrs Target | Scoped on demand | Ray-Ban Meta | 1920x1080 @ 30 FPS | Wide-angle camera lens intrinsics | Tier 1 · Tier 2 |
| Agriculture & Outdoor Tasks | Outdoor | 15,000 clips / 125 hrs Target | Scoped on demand | iPhone LiDAR | 1920x1080 @ 30 FPS | LiDAR Depth Maps & Stereo Depth Extrinsics | Tier 1 · Tier 2 · Tier 3 |
| Video RLHF Policy Annotations | RLHF | 2,000,000 comparisons Target | Scoped on demand | Human Feedback Portal | Varies (Commissioning Lab Ingestion) | Reward score, preference rankings, and failure rationales | Tier 1 · Tier 2 |
| Game Environment & Simulator Data | Simulation | 12,000 scenes / 10,000 hrs Target | Research phase | Simulator Engine | Synthetic RGB-D @ 60 FPS | Ground-truth joint angles, SLAM trajectory, and depth matrices | Tier 1 · Tier 2 |
Common to all eight
What every brief includes, whatever you commission
The specifications above differ in domain, device, volume and tier. Everything in this section is identical across all eight, so it is written here once and linked from each brief rather than repeated on every one.
How a brief is scoped, and what "on demand" means
Six of the eight briefs below read scoped on demand: for each, we would scope the brief, recruit for it and quote it as one engagement.
A pilot dataset is produced when a Commissioning lab asks for one. A brief here is the specification that dataset is produced against. One brief reads pipeline validated: a recording from that task family has been through the pipeline end to end and is published in full with its checksum, and the badge refers to the pipeline, not to a rig or a hub. One reads research phase: it is under evaluation and not open for commission.
Three annotation tiers, one schema
Every brief is annotated against FH-Ego v1 and against nothing else. Tier 1 is the task and session record, Tier 2 is the action segment, and Tier 3 is the timeline tracks that carry the kinematics. The Annotation column above says which tiers a brief includes.
The exporter offers no custom field set. It emits one schema, and every field of it is published with its shape, its unit, its sampling rate and the tier it belongs to: FH-Ego v1, field by field.
Five delivery containers, and two we decline
The same annotated record is written into any of five containers, each with a writer in this repository and each read back off disk by its own verifier before it ships. Two more are declined.
Which five, which two, and the reasoning for each: what arrives, and in what container.
A consent receipt, written before the first frame is uploaded
Every recording is consented on the Capture Partner's own phone before capture, and the receipt is written before the upload begins. A receipt is eleven fields hashed into a chain, each entry carrying the hash of the one before it, so a receipt cannot be edited after the fact without breaking every entry that follows.
Consent is granted over the recording, not over a clip, and it can be withdrawn. A withdrawal makes byte-identical footage undeliverable — packaging refuses rather than skips.
You can test a receipt yourself, in your own browser, against the published chain: check a consent receipt. How the chain is built and what its digest proves: the consent ledger.
Faces blurred at ingestion
Faces the detector finds are blurred when footage arrives, before any person here opens it. If redaction cannot run, the clip stops there.
Audio is removed rather than blurred: what a Commissioning lab receives carries no audio track, and that is checked rather than assumed — the encoder writes video only, and the stage reads its own written file back and raises if it finds sound.
How redaction runs, and what happens when it cannot: how faces are blurred.
Nine reasons a clip can be rejected, and the partner is told which
Every clip is accepted, rejected or held. A rejection carries one or more reasons from a fixed list of nine, and the Capture Partner reads the same reason we recorded, in the same words, with guidance on what to do differently.
How a clip is judged, and the nine reasons in full: how a clip is judged, and what a partner is told.
What has to happen before a sample appears
A sample on a brief page needs a commissioned brief and a Capture Partner cohort recording against it. One recording has been through the pipeline end to end: it belongs to precision factory assembly, and it is published in full, annotation and checksum, on the reference implementation.
The stages a recording passes
Five stages are implemented. Each Clip stops before TRACK until a person releases its cohort by name, so the last 3 stages run only on released work.
| Stage | What it does | Today |
|---|---|---|
BLUR | Faces detected and blurred, before any person here opens the footage. | Runs today |
SEGMENT | The continuous upload cut into task-bounded clips, and each clip judged. | Runs today |
TRACK | Hand landmarks read per sampled frame. | Parked |
CAPTION | A caption written for each action segment. | Parked |
EXPORT | The annotated record written into the delivery container. | Parked |
Every volume above is a target
Each row is a volume a brief would commit to, agreed before capture and carrying a Target badge wherever it is printed.
The pipeline has run end to end on 1 recording, published in full with its annotation and its checksum: the reference implementation.
The refusals
What a delivery does not include
Assembly refuses rather than skips, and every refusal names what caused it. These are conditions the code checks when a package is built, not undertakings we give.
Footage nobody consented to
Every clip's consent receipt is re-checked at the moment a package is assembled, not at the moment the release was cut. A partner who withdraws after a release is frozen still stops that delivery — byte-identical footage becomes undeliverable, and the package refuses rather than quietly shipping without it.
You can test any receipt against the published chain in your own browser: check a consent receipt.
Footage that skipped redaction
Faces the detector finds are blurred at ingestion, before any person here opens the footage and before anything downstream runs. If redaction cannot run, the pipeline stops — there is no path that emits a clip the redaction pass could not process. How faces are blurred.
Sound
The delivered clip carries no audio track, and the absence is checked rather than inherited: the encoder writes video only, and the stage reads its own written file back and raises if it finds one.
A package that does not verify as the package we named
A release that does not verify intact refuses. So does a clip with no exported annotation — a package with silently missing annotations is refused by name rather than shipped light. The delivery's own identifier is a hash over the sorted listing of every file inside it, each path paired with its own checksum, so rebuilding that listing is a test you can run on what arrives.
A container without its own reader
A format outside the published set refuses, and so does a format whose independent verifier fails on any clip's export. Every container we offer is read back off disk before it ships. What arrives, and in what container.
A verdict you have to take on trust
Every clip in a delivery carries its QA verdict and any rejection reasons in the compliance log, alongside its content hash, its consent receipt identifier and the status that receipt held at packaging time. The judgement travels with the data rather than staying with us. How a clip is judged.
A figure without its provenance
Every figure we publish carries its provenance and the command that recomputes it, and the record is annotated field by field: what each one is, its shape, its unit, how often it is sampled and which layer produced it.
FH-Ego v1, field by field · every number we publish, and where it comes from