Case study · Annotation

A university research lab clears a 3,400-episode backlog and moves its old labels to a new 14-action list.

Client: University research lab
Footage: 120 hours of two-arm teleoperation, 3 cameras each
Tasks: 6 kitchen tasks
Labels: Hand and object tracks, action segments, task descriptions

episodes labelled: 3,400
weeks, vs a 9-month estimate: 5 wks
of audited boundaries within 0.2 s of a second annotator: 96%
of episodes sent back for rework: 2.1%

The situation

3,400 episodes over 18 months: 1,100 had one action label each, the rest had none. The lab's next policy needed every step segmented, on a new list of 14 actions.

What they asked for

Where does each step start and end, to within 0.2 s?
Which episodes include a second attempt at a step?
Can the 1,100 old labels move to the new action list, or must they be redone?

What Talo delivered

Tracks for both hands and every object, segments from the 14-action list and a task description, in the lab's existing dataset format.

episode 2207 · pour_water · 0 to 24 sleft arm, 1 of 2

Old label

pour

New labels

reach
grasp
retry
lift
pour
place
release

The second grasp is labelled "retry", so the lab can keep it or filter it out for each experiment.

820 old labels moved to the new list. 280 were redone.

Old labels	What happened	Episodes
Clear single action	Split into segments from the new list	820
Wrong or ambiguous	Labelled again from the video	280
No label	Labelled from scratch	2,300

How it ran

1
Rules, week 1
Talo and the lab wrote a definition for each of the 14 actions, with the cue that starts and ends it and two sample clips.
2
Pilot, 50 episodes
The lab reviewed the pilot batch and changed three rules, including where "lift" ends and "pour" begins.
3
Weekly batches
Two passes per episode. A second annotator audited 10% of each batch, and the lab spot-checked every delivery.

Outcome

Returned episodes came back within the same week. The written rules can be cited in the paper.

The automatic first pass, on a public dataset

Episode 0 of Unitree's open G1 pouring dataset: a humanoid with two dexterous hands, driven by a teleoperator. The chart shows two signals from the robot's joint data. Move across it, pick a segment, or play the episode.

Segment	Time
	0.0 to 2.7 s
	2.7 to 3.4 s
	3.4 to 4.7 s
	4.7 to 9.2 s
	9.2 to 12.3 s
	12.3 to 12.6 s
	12.6 to 14.9 s

Segments here are an automatic first pass from the robot's own signals, coarser than a delivery, where an annotator checks every boundary against the video. Source: unitreerobotics/G1_Dex3_Pouring_Dataset, episode 0, Apache-2.0. Video trimmed and scaled.

Across all 311 episodes (68 minutes)

All seven segments found automatically295
A segment the automatic pass missed; sent to a person16
Two grasps in one episode: retry candidates2

The automatic pass does most of the work. The flagged episodes are where an annotator's time goes.

Have a backlog of recordings? We label a small batch of your own footage first.

Request an annotation pilot Next: household collection →