Video World Models Need Predictable Latents, New arXiv Paper Argues

A new paper from Mattia Scardecchia and Sherry Yang proposes that video world models can be significantly improved by optimizing their latent spaces for temporal predictability — without sacrificing accurate pixel-space reconstruction. The key insight is that pairing temporal objectives for representation learning with a past-conditioned decoder architecture yields compounding gains: up to 5x cheaper inference, over 7x faster training, and 15–24% lower pixel and perceptual errors on out-of-distribution and long-horizon tasks.

The work directly targets a persistent tension in generative video modeling: latent spaces trained for reconstruction are not inherently suited to dynamics prediction, while latents designed for planning often lose fidelity to real observations. By reconciling both objectives, the authors also report that their world models align more closely with real-world policy evaluation when a vision-language model is used to judge task success from pixel-space outputs — a result with practical implications for robot learning and simulation-based policy development.