A robotics startup gets household demonstrations from 30 homes it could not reach.

The situation

The startup trains household manipulation policies. Its four prototype robots live in one test house, so its own teleoperation data showed one kitchen, one set of plates and one kind of light. Policies that worked there failed in kitchens they had not seen.

It needed the same tasks recorded in many homes, without shipping robots to each one.

What they asked for

• 20 tasks, from loading a dishwasher to folding towels, each recorded in at least 25 homes.
• Gripper demonstrations with fingers that match its robot's gripper, so the robot can learn from them directly.
• Every episode labelled to the same rules as its in-house data.
• Bad episodes caught before delivery, not after training.

What Talo delivered

Recording partners captured 4,100 head-camera episodes and 2,100 UMIUniversal Manipulation Interface: a handheld gripper with a wrist camera, used to record demonstrations without a robot. gripper episodes. For UMI episodes, the gripper's position, orientation and opening width were recorded at 60 Hz and aligned to each video frame. Talo labelled every episode with object tracks, action segments and a task description.

episode u-0918 · place_cup · home 17 · 0 to 8 sUMI

Video

Gripper width
(open at top)

Segments

1. reach
2. grasp
3. lift
4. place
5. release
6. retract

0 s4 s8 s

Segment boundaries line up with the gripper stream, so "grasp" ends where the width stops closing.

How it ran

1. 1

   Task list

   Each of the 20 tasks was written as a short script: start state, end state and what may vary between homes.

2. 2

   Pilot week

   3 tasks in 4 homes, 180 episodes. The startup loaded them into its training code to confirm the gripper data matched its robot.

3. 3

   Weekly batches

   About 1,000 episodes a week. Each passed an intake checkA check on each new recording before labelling: camera clear, every data stream present, task completed., two labelling passes and a 10% auditA second annotator labels a sample of episodes on their own, and the two sets of labels are compared. by a second annotator.

Outcome

Every episode shipped with its rules and check log, so the startup could mix it with its own teleoperation data.

The automatic first pass, on a public dataset

Episode 0 of the open UMI cup dataset: a person carries a handheld gripper and places a cup on its saucer, outdoors. The gripper records its opening width with every frame. Move across the chart, pick a segment, or play the episode.

Gripper close-and-open cycles per episode, across all 1,447 episodes (19.4 hours)

• None found (public data)49
• One1,011
• Two354
• Three30
• Four or more3

The task needs one grasp. About one episode in four shows more than one, which could be a regrasp, a slip or a noisy reading. Those are the episodes to check before training.