Talo Request a pilot
← Back to annotation

Case study · Annotation

A university research lab clears a 3,400-episode backlog and moves its old labels to a new 14-action list.

Client
University research lab
Footage
120 hours of two-arm teleoperation, 3 cameras each
Tasks
6 kitchen tasks
Labels
Hand and object tracks, action segments, task descriptions
episodes labelled
3,400
weeks, vs a 9-month estimate
5 wks
of auditedA second annotator labels a sample of episodes on their own, and the two sets of labels are compared. boundaries within 0.2 s of a second annotator
96%
of episodes sent back for rework
2.1%

The situation

The lab records two-arm teleoperation on six kitchen tasks. Over 18 months it built up 3,400 episodes. Students had labelled 1,100 of them with a single action per episode. The rest had no labels.

The lab's next policy predicts the current step before acting, so every episode needed action segments. The old labels also had to move to a new list of 14 actions.

What they asked for

  • Where does each step start and end, to within 0.2 s?
  • Which episodes include a second attempt at a step?
  • Can the 1,100 old labels move to the new action list, or must they be redone?

What Talo delivered

Each episode came back with tracks for both hands and every object handled, segments from the 14-action list, and a one-line task description. Labels were written into the lab's existing dataset format, so nothing needed converting.

episode 2207 · pour_water · 0 to 24 sleft arm, 1 of 2

Old label

  1. pour

New labels

  1. reach
  2. grasp
  3. retry
  4. lift
  5. pour
  6. place
  7. release

The second grasp is labelled "retryA second attempt at the same step after the first one failed, such as a missed grasp.", so the lab can keep it or filter it out for each experiment.

820 old labels moved to the new list. 280 were redone.

Old labelsWhat happenedEpisodes
Clear single actionSplit into segments from the new list820
Wrong or ambiguousLabelled again from the video280
No labelLabelled from scratch2,300

How it ran

  1. 1

    Rules, week 1

    Talo and the lab wrote a definition for each of the 14 actions, with the cue that starts and ends it and two sample clips.

  2. 2

    Pilot, 50 episodes

    The lab reviewed the pilot batch and changed three rules, including where "lift" ends and "pour" begins.

  3. 3

    Weekly batches

    Two passes per episode. A second annotator audited 10% of each batch, and the lab spot-checked every delivery.

Outcome

Returned episodes came back within the same week. The written rules can be cited in the paper, and future recordings are labelled the same way.

The automatic first pass, on a public dataset

Episode 0 of Unitree's open G1 pouring dataset: a humanoid with two dexterous hands, driven by a teleoperator. The chart shows two signals from the robot's joint data. Move across it, pick a segment, or play the episode.

SegmentTime
0.0 to 2.7 s
2.7 to 3.4 s
3.4 to 4.7 s
4.7 to 9.2 s
9.2 to 12.3 s
12.3 to 12.6 s
12.6 to 14.9 s
Segments here are an automatic first pass from the robot's own signals, coarser than a delivery, where an annotator checks every boundary against the video. Source: unitreerobotics/G1_Dex3_Pouring_Dataset, episode 0, Apache-2.0. Video trimmed and scaled.

Across all 311 episodes (68 minutes)

  • All seven segments found automatically295
  • A segment the automatic pass missed; sent to a person16
  • Two grasps in one episode: retry candidates2

The automatic pass does most of the work. The flagged episodes are where an annotator's time goes.

Have a backlog of recordings? We label a small batch of your own footage first.

Request an annotation pilot Next: household collection →