World models
LiveFoley: real-time sound for AI and game video
LiveFoley is a research model that generates synchronised sound effects from live video with a 0.08-second look-ahead. It is built to be a sound engine for games, AR/VR and world models whose own audio cannot be reused.
A research team has published LiveFoley, a model that watches video as it is produced and generates matching sound effects on the fly. The paper went onto arXiv on 4 November 2026. It targets a gap in interactive worlds: sound that keeps up with what the player does.
What LiveFoley does
LiveFoley is a streaming video-to-audio model. It takes a video feed frame by frame and writes the audio for it with no pause for the clip to finish. The authors describe it as a causal autoregressive diffusion architecture, meaning it only uses what it has already seen plus a very short look-ahead.
Earlier streaming systems cut latency by shrinking the window of video they look at, which made sound drift out of sync. LiveFoley instead adds a “Trailing Sliding Window” that turns an ordinary segment-level video encoder into a streaming one without retraining. A configurable hop size sets the trade-off between latency and compute.
The audio side widens the denoising window of its diffusion head. The default, History LocDiT, adds past context at no latency cost. An alternative, Future-Warmup LocDiT, uses an approximate future condition for better quality in exchange for a little extra look-ahead.
The numbers
- Look-ahead: 0.08 seconds at the visual frontend, which the paper says is below the human threshold for noticing delay.
- Alignment: a DeSync score of 0.48 on the VGGSound test set for the 48k model, against 0.47 for the offline MMAudio-L and 0.67 to 1.03 for the streaming baselines.
- Quality: the best Fréchet distance scores among streaming systems on the same test.
- Listening test: six participants rated it above the other online systems for quality (3.23) and alignment (3.58), and close to the offline MMAudio-L on alignment (3.58 against its 3.79).
- Hardware: end-to-end latency below the perceptual threshold on a single A800 GPU.
The authors’ own user study found people notice speech running late at about 0.49 seconds, and ambient sound while exploring at about 0.85 seconds.
Access and limits
The project page and arXiv paper are live. The GitHub repository says “Code coming soon”, so there are no public weights yet. The listening test is small, with six participants, and the benchmark uses short VGGSound clips. Real game footage over long sessions is the harder case, and the paper’s results should be read as a research benchmark, not a product claim.
Our take
Many world models generate pictures only, or tie their sound to their own video generator. The LiveFoley paper points out that this makes the audio unusable with game engines, AR/VR and live camera feeds. A separate model that listens to any video stream fixes that. For anyone building a generated world, it is a plausible route to footsteps, impacts and ambience that match the frame you are looking at. We explain why the visuals are the easy half in how AI world models work.