Nvidia's DreamDojo is an open source world model for robot training
Nvidia's AI research team has released DreamDojo, an open-source, interactive world model for robotics.
It takes robot motor controls and generates a simulated future in pixels; no engine, no meshes, no hand-authored dynamics are required. Jim Fan, Director of AI and Distinguished Scientist at Nvidia, calls it "Simulation 2.0."
DreamDojo learns from human video instead of robot data
Real-world robot learning is bottlenecked by time, wear, safety, and resets, Fan explains. DreamDojo tries to work around this by learning from humans first.
The model is pre-trained on 44,000 hours of first-person human video footage with zero robot-in-the-loop. So-called "latent actions," a unified representation inferred directly from videos, capture what changed between world states without knowing the underlying hardware, which lets the model train on any first-person video as if it came with motor commands attached.
In a second step, the model gets post-trained on a specific robot to fit its hardware. Fan describes it as separating "how the world looks and behaves" from "how this particular robot actuates." The base model learns general physical rules, then snaps onto the robot's unique mechanics.
A real-time version of DreamDojo runs at 10 frames per second, stable for over a minute of continuous rollout. It supports live VR teleoperation inside the dream, policy evaluation in the neural simulator, and model-based planning, all within the world model.
According to Fan, all weights, code, post-training dataset, eval set, and whitepaper are openly available. DreamDojo is built on Nvidia Cosmos, which is open-weight too. More details are on the project page and in the paper.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe nowRead on for the full picture.
Subscribe for hype-free coverage.
- Access to all THE DECODER articles.
- Read without distractions – no Google ads.
- Access to comments and community discussions.
- Weekly AI newsletter.
- 6 times a year: “AI Radar” – deep dives on key AI topics.
- Up to 25 % off on KI Pro online events.
- Access to our full ten-year archive.
- Get the latest AI news from The Decoder.