Nvidia turns to synthetic data to tackle robotics’ biggest challenge: the lack of training data.

"We call this the big data gap in robotics," a Nvidia researcher said at the Physical AI and Robotics Day during GTC Washington. While large language models train on trillions of internet tokens, robot models like Nvidia’s GR00T have access to only a few million hours of teleoperation data, gathered through complex manual effort - and most of it is narrowly task-specific.

Nvidia’s answer is to rethink what it calls the "data pyramid for robotics." At the top sit real-world data - small in quantity and expensive to collect. In the middle lies synthetic data from simulation - theoretically limitless. At the base is unstructured web data. "When synthetic data surpasses the web-scale data, that's when robots can truly learn to become generalized for every task," the team states. With Cosmos and Isaac Sim, Nvidia aims to turn robotics’ data shortage into a compute challenge instead.