Training a Robot Takes a Film Set and a Haystack of Video

Teaching a robot to do a simple job starts a lot like shooting a movie.

Eric Landau, the co-founder of Encord, a company that prepares data for physical AI, which means robots and self driving cars, says early robot companies use a facility his company runs in the Bay Area. There, staff build a set for a specific environment, have a robot perform a task in it, and record what happens. That staged practice gives companies that are earlier in development enough examples for the first round of training, what AI builders call pre-training.

A lot of the training material comes from production. Landau says his system ingests production data from customers, including video streams, sensor data, audio, and language. Its biggest growth now comes from robotics, including human-like robots, home robots, and factory robots, while self driving cars are further along and closer to wide use.

Then comes the work of sorting it. Teams must manage the data, select the right parts, add labels and extra detail, and test whether the robot is improving. Landau describes the selection job as "finding a million needles in a billion haystacks," because there is far more data than is needed and the useful parts must be picked out. The useful clips are rare compared with the total.

The scale is enormous. Landau says his company, which serves hundreds of customers, handles multiple petabytes of this kind of data, more, by his estimate, than was used to train GPT-4, the language model behind an earlier version of ChatGPT. A petabyte is about a million gigabytes.

That is why physical AI needs so much material from the real world. A chatbot can learn from writing already on the internet, but a robot must learn from what its cameras and sensors record in the physical world.