Nvidia's DreamDojo is an open source world model for robot training

Nvidia's AI research team has released DreamDojo, an open-source, interactive world model for robotics.

It takes robot motor controls and generates a simulated future in pixels; no engine, no meshes, no hand-authored dynamics are required. Jim Fan, Director of AI and Distinguished Scientist at Nvidia, calls it "Simulation 2.0."

DreamDojo learns from human video instead of robot data

Real-world robot learning is bottlenecked by time, wear, safety, and resets, Fan explains. DreamDojo tries to work around this by learning from humans first.

The model is pre-trained on 44,000 hours of first-person human video footage with zero robot-in-the-loop. So-called "latent actions," a unified representation inferred directly from videos, capture what changed between world states without knowing the underlying hardware, which lets the model train on any first-person video as if it came with motor commands attached.

In a second step, the model gets post-trained on a specific robot to fit its hardware. Fan describes it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model learns general physical rules, then snaps onto the robot's unique mechanics.

A real-time version of DreamDojo runs at 10 frames per second, stable for over a minute of continuous rollout. It supports live VR teleoperation inside the dream, policy evaluation in the neural simulator, and model-based planning, all within the world model.

According to Fan, all weights, code, post-training dataset, eval set, and whitepaper are openly available. DreamDojo is built on Nvidia Cosmos, which is open-weight too. More details are on the project page and in the paper.