World models
NVIDIA Cosmos 3 Sim: real-time interactive world models
NVIDIA has post-trained Cosmos 3 into three open streaming world models that respond to actions and spatial controls. The robot model runs at 48 frames a second at 480p on one GPU.
NVIDIA has published a technical report, code and three model checkpoints that turn its Cosmos 3 world foundation model into real-time, streaming interactive simulators. The paper was submitted to arXiv on 4 November 2026.
The aim is physical AI: training and testing robots and self-driving systems. But the method is the one games need too. A model receives controls, then generates the next stretch of video.
How it works
All three models come from one post-training recipe. New controls condition the next video chunk. The model denoises that chunk, appends the clean controls and video to a cached context, and repeats. NVIDIA calls the models “causal-distilled”, meaning they generate forward in time in small chunks rather than producing a whole clip at once.
The three models
- Cosmos3-Nano-Sim-Bimanual predicts what happens next from camera movement and robot arm actions. NVIDIA lists 48.1 frames a second with 166 ms latency, at 480p on a single GB300 GPU. It reports higher video quality than DreamDojo 2B and 14B on the AgiBot-World and AgiBot-GEAR datasets.
- Cosmos3-Nano-Sim-Transfer takes depth, segmentation, edge or blurred-video controls from a simulator and a text prompt, and produces RGB video. It runs at 15.85 frames a second with 240 ms latency, at 640 × 640 on one GB200 GPU. NVIDIA reports 59.2% higher edge F1 and 26.5% lower depth error than the Cosmos 3 reference.
- Cosmos3-Nano-Sim-Auto generates synchronised multi-view driving video: 7 RGB views plus LiDAR at 480p per view, at 16.7 frames a second. It needs 16 GB200 GPUs.
Access and limits
The Bimanual model card lists the licence as OpenMDW 1.1 and says the model is ready for commercial or non-commercial use. Code is on GitHub and checkpoints are on Hugging Face.
The hardware is data-centre class, not a games PC. Every figure above is NVIDIA’s own, and the demos cover robots, camera navigation and driving rather than games. In the Transfer model, a separate simulator still keeps physics and scene state, and the model only supplies the appearance.
Our take
That split matters for anyone building interactive worlds. The Transfer model treats a conventional simulator as the source of truth and uses the video model as the renderer. The Bimanual model lets the video model carry the dynamics itself. We covered the trade-off in how AI world models work: rules you can inspect on one side, flexible but drifting prediction on the other.
For game makers, the useful signal is speed and openness. A 166 ms response at 48 frames a second is the kind of latency a playable loop needs, though on hardware few players own. Open checkpoints let studios test it on their own scenes.