World models
How AI world models work, and why games care
A world model predicts what happens next in a scene, frame by frame, as you act. Here is how the main designs work, where they fail, and what they mean for anyone building interactive worlds.
A video model makes a clip and stops. A world model keeps going. You press a key, the model predicts the next moment, and the scene answers. Do that thirty times a second and you have something that feels less like a film and more like a place.
That is the whole idea, and it is why game developers have started paying attention. But the phrase “world model” covers several quite different designs, and they fail in different ways. This guide explains what a world model is, how the main approaches work, and where the hard problems still sit.
What is a world model?
A world model is a system that learns how a situation changes over time, so it can predict what comes next given what happened before and what an agent does now.
The idea is older than the current boom. Researchers in reinforcement learning have long trained small models to imagine the consequences of actions, so an agent can practise in its head. What changed recently is scale and output. Modern world models are large video-generating networks that produce full frames, and some can be steered live with a controller.
Odyssey, a lab founded by people who spent a decade on driverless cars, describes its latest model as “a learned dynamical system”: an autoregressive diffusion transformer that predicts how objects move and interact through space and how situations evolve. The driving background is telling. A car needs to predict the world to decide what to do next. A game needs to predict the world to decide what to show.
The core loop: predict the next chunk
Most interactive world models share one structure.
- Start from a seed. This is usually a text prompt, an image, or both.
- Take an action. A key press, a camera movement, or a text event such as “it starts to rain”.
- Predict the next chunk of video, conditioned on the seed, the recent frames and the action.
- Show it, then repeat.
The model is autoregressive: each chunk depends on the ones before it, never on the future. That one-way flow is what makes interaction possible. A model that needs to see the whole clip before drawing the first frame cannot respond to a player.
The generator underneath is usually a diffusion model, which starts from noise and refines it into an image over many steps. Many steps means slow. So the second ingredient is distillation: training a fast student model with only a few steps to copy a slow, careful teacher. The Streaming Code World Model release, for example, reports a four-step distilled model that is nine times faster than 35-step sampling.
Three ways to build a world
Here the designs diverge. The useful question to ask of any world model is: where does the state of the world live?
1. State lives in the pixels
The purest approach asks one network to remember everything, enforce the physics and draw every frame. What the world contains is only implied by what the recent frames looked like.
This is the design behind Odyssey-3 and WorldPlay2. It is flexible: you can generate a world from nothing but a sentence, with no engine and no assets. It is also where the field’s best-known failure comes from. If a tree leaves the view and the model forgets it, the tree is gone. Errors also accumulate over long runs, a problem researchers call drift.
The recent papers are largely attempts to fix this from the inside. The WorldPlay2 paper compresses past frames into compact memory tokens so a long session stays affordable and the model can still recall earlier scenes. It also splits control into separate channels: movement and camera on one side, and a description divided into scene, character and event on the other. Its authors argue that separating these makes control easier to learn (arXiv:2609.35560). For a designer, that is the right grammar: change the weather without resetting the world.
2. State lives in a game engine
The second approach takes the opposite view: do not ask a neural network to remember anything it can look up.
Skywork’s Matrix-Game 4.0 shows this clearly. According to the paper, a lightweight game engine maintains world state and executes interaction rules. A multimodal coding agent turns a player’s idea into game code and scene captions. A video model then repaints the engine’s low-poly proxy view into detailed visuals. The authors call the engine-plus-agent layer an executable game harness.
Because the engine owns geometry and rules, a tree you walked past is still there when you turn round: it never stopped existing in code. The neural model only has to make it look good. The payoff is speed. Skywork reports 480p at 60 frames per second on a single NVIDIA RTX 5090, with 112 ms from key press to display. Our Matrix-Game 4.0 write-up covers what is and is not released.
The trade-off is that someone has to write the rules, or an agent has to. You get reliability and lose some of the effortless open-endedness.
3. State lives in 3D objects
The third approach avoids both pixels and engines by representing the scene itself as structured objects. Viggle’s JST-2 models a scene as persistent “atoms”, each with an explicit position in 3D space plus learned attributes, and decodes them into images or 3D Gaussian splats. We covered the details in our Viggle JST-2 post.
The appeal is that objects keep their identity and can be queried and moved. The current limit, by Viggle’s own account, is that scenes are explorable static geometry rather than a playable, rule-bound world, and object interactions still rely on conventional simulation.
Between these three sit hybrids. The Westlake AGI Lab’s Code World Model has an agent update the world by writing executable code, while a renderer produces the frames. Its streaming variant generates video in chunks of about 0.71 seconds, but each chunk takes around 5.1 seconds on a single H200, so it is not real time yet.
Why speed is the real constraint
Quality gets the headlines. Latency decides whether you have a game.
A film tolerates a render that takes a minute per second of footage. A player does not tolerate a delay between pressing a key and seeing a result. Everything about the design above is shaped by that budget: autoregressive chunks, few-step distillation, compressed memory, engine-owned state.
The numbers show how fast the field is moving, and how uneven it is. SolarX’s open SolarWM H3 Realtime work reports cutting time to the first responding frame from about 6.9 seconds to 0.58 seconds through systems engineering: compiling the model, changing the attention backend, reusing cached history and replaying CUDA graphs. Their median key-to-screen latency in a web demo is still 1.15 seconds. That is fine for exploring a scene and well short of a game engine’s instant response. These are the authors’ own figures, and they are best read as relative speed-ups.
How do you measure a world model?
Visual quality is easy to judge and easy to fake. Physical plausibility is harder.
Benchmarks such as Physics-IQ ask a model to continue videos of real physical experiments, covering fluids, optics, solid mechanics, magnetism and thermodynamics, and then compare the prediction with what really happened. Odyssey reports that its Pro model scores 66.1 on the Physics-IQ Verified video-to-video benchmark, and presents that as the highest reported score. That figure uses best-of-several sampling. Independent coverage measured a lower single-generation number, so check how any score was produced before comparing it.
For game makers, the questions that matter are less academic:
- Does the world stay consistent when I look away and back?
- Does it respond within a tenth of a second or so?
- Can I direct it, rather than just watch it?
- Does it keep going for an hour without drifting into nonsense?
No model answers all four today. Long-session memory in particular remains the open problem across the field.
What world models cannot do yet
Be sceptical of any demo that looks like a finished game.
- They forget. Memory compression helps, but a world that reliably recalls what you did an hour ago is still unsolved.
- They are expensive. WorldPlay2’s hosted build streams at about 17 frames per second on four datacentre GPUs, and the weights are around 82 GB. Matrix-Game 4.0 is the exception that runs on a consumer card.
- They are not designed. A model can produce a believable street. It does not know your story, your characters’ motives, or why this door matters. That layer is authored.
- Licences vary. Some of the most interesting releases are research-only. WorldPlay2 is CC BY-NC 4.0, so it cannot be used commercially.
What this means for people building interactive worlds
The sensible reading is not that world models replace engines. It is that the work is splitting into two layers: something that keeps the truth of the world, and something that performs it.
Performance, meaning how the world looks and moves, is improving quickly and getting cheaper. The truth layer is the harder and more valuable one: canon, characters with their own state, consequences that persist, a story that holds together across sessions. That is the argument we make in AI models as game generators, and the recent engine-plus-renderer designs bear it out.
It also connects to a larger idea. A world that models itself can run without you, which is where simulated worlds become a storytelling medium rather than a backdrop. The studios best placed are the ones that already think in terms of state, rules and canon. The visuals are about to become the easier part.
