FH-Ego v1, field by field

Read this before the tables

Tier 1 · Task / session

The clip

One object per delivered clip. Everything else hangs off it.

Inside a clip

The outlined panel marks where a clip's video goes. The labels set out what its FH-Ego v1 annotation records.

Each label lists the fields that hold its signal. Measured values are read from the video file. Derived values are produced by a model or a rule.

task_id — Internal identifier for the annotation run

title — Human-readable title for the clip

task_label — Taxonomy code for the task performed

task_name — Readable name of the task performed

details — Free-text description of the session

scene — Environment the capture was made in

usable — Whether the clip passed QA and may be delivered

start_frame — First frame of the clip

end_frame — The clip's frame count. Frames are numbered from zero, so the last frame is end_frame minus one: on the reference clip end_frame is 1843 and the final action window ends at 1842.

fps — Frames per second of the source video

actions — The task-bounded action segments this clip was cut into

tracks — The parallel supervision layers

provenance — Where this clip came from — see the provenance fields below

Tier 1 · Task / session

Where the clip came from

Carried inside the annotation. The hash lets you verify the bytes offline by rehashing them; the consent receipt is the one handle that resolves against our API rather than in your hand.

clip_id — Canonical FH1 clip identifier; the tail is derived from the clip's SHA-256

scheme_version — Identifier scheme version

content_sha256 — SHA-256 of the capture. Rehash the file to verify this annotation describes the bytes you hold

cohort — Capture cohort

captured_at — When the capture was recorded (partner-declared)

consent_receipt_id — Consent ledger receipt covering this clip; verifiable on request

qa_verdict — Automated QA outcome for this clip

qa_reject_reasons — Closed-vocabulary reject reasons, empty when accepted

derived_fields — Fields on this record that are model-generated, never measured (AGENTS.md 4.1)

caption_source — Which captioner produced the derived caption fields: a model id (e.g. claude-opus-5) or 'heuristic_stub' for development filler. CARRY C.6

Tier 2 · Action segment

One action segment

The clip is divided into fixed windows, and each window is given a verb-noun code. The windows are derived: their length follows from the frame rate, not from what happens in the footage.

start_frame — First frame index of action chunk

end_frame — Last frame index of action chunk

hand — Operative hand: 'left', 'right', or 'both'

scene_semantic — Interacting surface/workstation area

camera_motion — Camera motion classification

action — Standardized verb-noun action code (e.g. scoop_food)

description — Dense natural language sentence describing physical interaction

Tier 3 · Timeline track

The parallel tracks

Supervision layers that run the length of the clip alongside each other.

hand_control — Header describing the kinematics track

object_interactions — The noun half of each action code, de-duplicated. Read from the action codes, not from object detection; no object detector runs in the pipeline. Derived, and listed in the clip's derived-fields list.

action_verbs — Distinct action codes in the clip. Derived, and listed in the clip's derived-fields list. When the caption source is a stub, they come from a fixed pool of five codes.

dense_captions — One caption per action window. Derived, and listed in the clip's derived-fields list. The caption-source field names what wrote them; a stub value means a template, not a model.

kinematics_3d — The sampled wrist track, one entry per sampled frame per detected hand

Tier 3 · Timeline track

The kinematics track header

Describes the sampled track below it — including the sample interval, so the file states its own rate.

hand_dominance — Distinct operative-hand codes across this clip's action windows: 'left', 'right' or 'both'. Computed from the action track, not from the hand tracker, so it lists hand codes rather than detected hands.

total_frames_annotated — Frames in the source clip

kinematic_sample_frames — The number of entries in kinematics_3d: one per detected hand per sampled frame, so two hands in view give two entries for one frame. On the reference clip: 294 entries across 185 distinct frames.

kinematic_sample_every_n_frames — Every Nth frame was sampled into kinematics_3d.

kinematic_sample_rate_hz — Derived: clip fps ÷ kinematic_sample_every_n_frames.

Tier 3 · Timeline track

One kinematics sample

One entry per sampled frame per detected hand. This is the layer that carries the detector's output.

frame_idx — Index of the source video frame this sample was taken from

timestamp_sec — Time of this sample from the start of the clip

hand — Which hand this sample describes, set by where the wrist sits in the frame: with two or more hands in view the leftmost is left, and with one hand left of centre is left. The detector's own left-or-right score is in confidence.

position_3d_m — Position of the wrist. x and y are its place in the frame, rescaled by fixed constants; z is a fixed offset plus the hand model's depth for the wrist. No physical unit: the values are not metres, and no camera intrinsics are published.

velocity_3d_mps — The change in position_3d_m since a hand with the same left or right label was last detected, multiplied by the frame rate. Tracking runs on every frame, before sampling. No physical unit: position's units per second, not m/s.

confidence — The hand detector's score, from 0.5 to 1, for its own left-or-right call on this hand. One value per detected hand; the 21 keypoints carry no score of their own. The hand label is set by wrist position, not by this call.

keypoints_21 — 21 MediaPipe hand landmarks, normalised image coordinates [x, y, z] per joint, in MediaPipe's canonical joint order (0 = wrist, 4 = thumb tip, 8 = index tip, … 20 = pinky tip)

What one real clip holds

Numbers below are counted from the published reference clip at build time, not typed.

The sample interval is recorded on every clip rather than assumed, so a file can always be checked against the rate it claims. A track sampled every tenth frame of thirty is a three-hertz track, and calling it thirty would be the easiest false number on this page to publish.

Open the clip these numbers come from · What arrives, and in what container · Every number we publish, and where it comes from

curl -sO https://fourthuman.ai/SHA256SUMS
curl -sO https://fourthuman.ai/reference-run.json
sha256sum -c SHA256SUMS        # or on a Mac: shasum -a 256 -c SHA256SUMS