What interactive video world models actually do on one GPU (and a tool to check)

Interactive video "world models" are having a moment — Matrix-Game, HY-World, and others generate playable worlds frame by frame, conditioned on your actions. Almost every launch quotes a frame rate. Almost none name the hardware behind it.

So I built a small open tool, LiveFrame, that measures what a model actually delivers on the hardware you run it on, and binds every number to a digest ledger anyone can re-verify.

Here's what three published models did when I measured them on a single GPU:

Model	Claim (as published)	Measured	Why they differ
Matrix-Game 2.0	25 fps (1× H100)	19.7 fps	same H100 — consistent with the claim
ABot-World-0	16 fps (1× RTX 5090)	5.85 fps	same RTX 5090; optional fp8/sage stack off
Oasis-500M	20 fps (proprietary engine)	1.06 fps	open weights vanilla (A100 PCIe) — different artifact
HY-World 1.5	24 fps (streaming; unstated)	1.36 fps	released pipeline runs batch (8× A100)

This isn't a "gotcha." The vendors aren't wrong — their numbers assume clusters. The point is that the number you can act on depends entirely on the configuration, and that configuration is usually missing from the headline.

Why a single-GPU number matters even if you deploy on a cluster

LiveFrame measures any config — it times frames as they're delivered regardless of how the model uses hardware internally, so you can point it at your 8-GPU setup and it'll time that. I report single-GPU for a practical reason: world models are per-user. One player → one live world. You can't cheaply shard a single interactive stream across a cluster (latency and sync get in the way), so GPUs-per-concurrent-stream is the dominant cost, and single-GPU throughput is roughly your cost per user. The cluster figure tells you it's possible; the per-GPU figure tells you what it costs.

Why you can trust the numbers

Three design choices do the heavy lifting:

It errs low. LiveFrame times frames as they're delivered to the host and counts native decoded frames only — never interpolated or duplicated ones. It structurally cannot inflate a number; if anything it under-reports.
Receipts. Every run writes a self-contained report.html, a machine-readable report.json, and a digest-bound ledger.json. liveframe world verify <run>/ledger.json re-checks every artifact byte — change one byte and verification fails. That's what makes a report you didn't produce trustworthy.
Prove it on CPU first. A synthetic backend ships in the package, so you can exercise the whole pipeline in seconds before touching weights.

How this fits with existing benchmarks

To be clear about prior art: there's a healthy field of world-model benchmarks — VBench, WorldScore, WBench, WorldExam — and they're good at what they do: evaluating quality (controllability, visual fidelity, physics, multi-turn consistency). VBench even reports a single-GPU efficiency number. LiveFrame isn't the first tool to time a model, and it doesn't try to score quality.

What it adds is the complementary piece those benchmarks don't cover: a re-verifiable, bring-your-own-model runtime harness for interactive models — runtime, control, stability, and cost, measured on your config, with a ledger anyone can re-check. It even exports to the worldmark/wbench formats, so it feeds the quality benchmarks rather than competing with them. Quality benchmarks tell you if a model is good; LiveFrame tells you if it's fast enough and what it costs.

Try it in two minutes

pip install "liveframe[world]" # Prove the loop end to end on CPU (no GPU, no model): liveframe world run synthetic --suite smoke # Re-verify any finished run, byte for byte: liveframe world verify runs/synthetic/smoke/<timestamp>/ledger.json # -> "verified": true

Bring your own model

Any model becomes a FILE.py:ClassName adapter — five methods and a declared-capabilities object. The harness treats it exactly like a real model: it validates every frame against what you declared, times every chunk, and fails the case (not the run) if the contract breaks.

Here's a complete, runnable stand-in:

# my_world.py from __future__ import annotations import time from collections.abc import Iterator, Sequence import numpy as np from liveframe.engine import NativeFrame from liveframe.world.backend import WorldBackendError, WorldCapabilities class MyWorld: def __init__(self, *, height=144, width=256): self._h, self._w = height, width self._canvas, self._pos, self._emission, self._loaded = None, np.zeros(2), 0, False @property def capabilities(self) -> WorldCapabilities: return WorldCapabilities( name="my-world", interaction_mode="interactive", # step() is called per chunk  native_fps=16.0, # the rate it's DESIGNED to be viewed at  chunk_frames=4, # pixel frames emitted per step()  resolution=(self._h, self._w), # exact; any other shape fails the case  # Declare EVERY action the suites command, or the case fails with a clear error:  actions=frozenset({"W", "S", "A", "D", "L", "R", "U", "N", "0"}), action_magnitudes={"W": 3.0, "S": 3.0, "A": 3.0, "D": 3.0}, supports_seed=True, ) def load(self): # load weights here; cold start is reported as context, not generation time  self._loaded = True def reset(self, image, prompt, seed): if not self._loaded: raise WorldBackendError("reset() before load()") self._canvas = np.tile(np.asarray(image, dtype=np.uint8), (3, 3, 1)) self._pos, self._emission = np.array([float(self._h), float(self._w)]), 0 self._rng = np.random.default_rng(seed) def step(self, actions: Sequence[str]) -> Iterator[NativeFrame]: delta = {"W": (-3., 0.), "S": (3., 0.), "A": (0., -3.), "D": (0., 3.), "L": (0., -1.), "R": (0., 1.), "U": (-1., 0.), "N": (1., 0.), "0": (0., 0.)} frames = [] for _ in range(self.capabilities.chunk_frames): for token in actions: self._pos += delta[token] self._pos[0] = float(np.clip(self._pos[0], 0, self._h * 2)) self._pos[1] = float(np.clip(self._pos[1], 0, self._w * 2)) r, c = int(self._pos[0]), int(self._pos[1]) frames.append(np.ascontiguousarray(self._canvas[r:r+self._h, c:c+self._w])) # Emit one NativeFrame per step — never buffer and dump at the end.  yield NativeFrame(index=self._emission, payload=np.stack(frames), native_frame_count=len(frames), metadata={"worker_ts": time.perf_counter()}) self._emission += 1 def run_script(self, script): # batch-mode models implement this; interactive ones raise  raise WorldBackendError("my-world is interactive; the harness calls step()") def close(self): self._canvas = None

liveframe world run my_world.py:MyWorld --suite quick # Or run it inside your model's own environment (its torch/CUDA): liveframe world run my_world.py:MyWorld --suite quick --python /path/to/model-env/bin/python

The three rules the harness enforces: capabilities must tell the truth (resolution is exact,every commanded action must be declared); step must emit work as it happens (no buffering tolook faster); and native frames only (no interpolation). Break any one and it fails the case with a plain-English reason.

Limitations, up front

smoke/quick suites are short, so the real-time verdict reports null on purpose — the real-time rule needs a continuous 60-second stream (the soak suite).
MG3/HY ran the short quick suite; HY ran batch + offloaded; MG3 on SDPA, not flash-attn.
Forcing cluster-designed models onto one GPU with offloading isn't their intended config, it's a "can it run on commodity hardware" number. The fully fair comparison measures the vendor's config on its cluster and compares cost-per-stream. I haven't funded that yet.

All of this is recorded in the run bundles, not buried.

Backstory

This grew out of a month-long experiment trying to hit real-time diffusion video on a MacBook.
I came up about 11× short and published exactly where the time went instead of hiding it. The arXiv write-up got rejected 🙂 but the useful thing turned out to be the measurement method,so I kept building. Thanks to the Wan, Causal Forcing, Matrix-Game, HY-World, and MLX teams whose work this measures.