Runway Characters: a 57B video model that talks in real time

Runway has published a research paper on the system behind Runway Characters, its video characters that hold a spoken conversation with a user. The paper appeared on arXiv on 3 November, and Runway’s own write-up followed on 5 November.

The headline is a serving problem, not a new model. Runway Characters runs an autoregressive video diffusion transformer of about 57 billion parameters, at 704 by 1088 pixels and 24 frames per second, for one user at a time.

Why real time is hard

A normal video model works on its own schedule: you send a prompt, wait, and get a clip. A character in conversation cannot fall behind. At 24 fps the system must produce a new four-frame chunk every 166.7 milliseconds, for as long as the conversation lasts. The player holds only a few chunks in reserve, and each frame carries its slice of the audio, so one late chunk freezes the face and stalls the voice together.

The numbers

• Characters launched on eight H100 GPUs per session. On four, the launch software took 240.9 ms per diffusion step, well over the 166.7 ms budget.
• Runway’s optimisations cut that to 159.3 ms (1.51 times faster, 24.5 fps sustained), so a session now runs in real time on four H100s.
• A fused communication kernel takes it to 147.4 ms (1.63 times faster). Runway says it rolled this kernel back once in production and has since re-enabled it.
• On three B200 GPUs a step takes 154.3 ms.
• Over seven weeks of production, the worker averaged above 23.5 fps in 99.5% of sessions that streamed for at least a second.
• The video pipeline takes 0.49 seconds of a 2.1 second median conversational turn.

Runway attributes much of the saving to moving repeated cache work to write time: 29% of the original step was whole-cache work repeated in every attention call. It also gives the decoder that turns latents into frames its own GPU. The paper reports no detected loss in lip-sync scores.

Runway also says that on the same GPUs, the two open-source real-time video systems it benchmarked either miss real time or add over a second of latency, at a third to half of its resolution.

Limits

This is a paper about infrastructure. It does not publish model weights, a price or a new API; the only open-source parts are two supporting libraries, AVTensor and confingy. Four H100s for a single conversation is a heavy cost per user, and the paper says more GPUs help the model but not the decoder: on eight H100s a step still takes 128.7 ms (31 fps) because the single decoder GPU cannot keep up.

Our take

For interactive worlds, the useful lesson is the budget. A talking character that never stops generating needs a fixed slice of hardware for every player, which is a different cost model from a pre-rendered cutscene. That is why we think the near-term use is a small number of hero characters rather than a whole cast, the same question we raise in what are generative NPCs. It also bears on generative interactive film, where a live character could answer the viewer directly. Expect the compute figure to fall, but plan for it now.

---

Sources

• Runway Research: Real-Time Video at Batch Size One
• arXiv: Runway Characters: Serving a 57B Video Diffusion Transformer for Real-Time Conversation
• Runway: Building Runway Characters