Video World Models

Why Does This Exist?

Historically, if you wanted an AI to understand physics (e.g., that a ball bounces, or that a cup breaks when dropped), you had to write complex rules using a physics engine (like Unity or Unreal). This is rigid and doesn't scale to the infinite complexity of the real world.

Video World Models take a radically different approach: unsupervised learning. Just as LLMs learn grammar and logic simply by trying to predict the next word in a text corpus, Video World Models learn intuitive physics, geometry, and object permanence simply by trying to predict the next frame in a massive dataset of real-world videos. If a model can perfectly generate the next 10 seconds of a video showing a car driving down a bumpy road, it must have developed a strong internal representation of 3D space, momentum, and lighting.

Think of It Like This

How It Actually Works

Video World Models (like OpenAI's Sora or Runway's Gen-3) are essentially highly advanced video generation models, but their utility goes far beyond making pretty clips.

1. Spatiotemporal Patches

Just as text models break sentences into tokens, video models break videos into "spatiotemporal patches." A video is just a 3D cube of data (Width x Height x Time). The model slices this cube into small 3D blocks. This allows the model to process variable resolutions, aspect ratios, and durations in a unified way.

2. The Transformer Backbone

Once the video is tokenized into patches, a massive Transformer architecture (often combined with a Diffusion process, creating a Diffusion Transformer or DiT) is trained to denoise or predict these patches.

3. Emergent Physics

The magic happens at scale. When you train a large enough DiT on enough video data, the model stops memorizing pixel patterns and starts learning the underlying latent rules that govern those pixels. Researchers have found that these models exhibit emergent capabilities:

• 3D Consistency: As a camera pans in the generated video, objects rotate in mathematically correct 3D space, even though the model has no explicit 3D engine.
• Object Permanence: If an object is occluded by another object, the model "remembers" it is there and it reappears correctly when the occlusion moves.
• Interactivity Simulation: The models can simulate the effects of actions, like a paintbrush leaving a stroke on a canvas, or a glass shattering.

The Limitation: "Dream Physics"

While impressive, these models do not actually calculate physics; they approximate them. This leads to surreal hallucinations (sometimes called "dream physics"). A generated person might walk through a wall, a chair might morph into a table, or a glass might shatter in slow motion while the water inside it remains solid. The model's understanding is statistical, not causal.

Show Me the Code

You don't typically write a Video World Model from scratch. Instead, you interact with them by conditioning their generation. Here is how you might condition a video model to simulate the future based on an initial frame.

1# Conceptual pseudo-code for interacting with a Video World Model API2from video_world_model_api import VideoPredictor3
4predictor = VideoPredictor(model="sora-v1")5
6# 1. Provide the current state of the world (an image)7current_state_image = "car_approaching_puddle.jpg"8
9# 2. Provide the "action" or condition as a text prompt10action_prompt = "The car drives through the puddle at high speed, splashing water."11
12# 3. The model simulates the physics of the water and the car13predicted_future_video = predictor.generate_video(14    initial_frame=current_state_image,15    prompt=action_prompt,16    duration_seconds=5,17    fps=3018)19
20# You now have a 5-second simulation of the physics event, 21# generated entirely via unsupervised learned representations.

Watch Out For

The Quick Version

• Video World Models learn the rules of the physical world by predicting video frames, similar to how LLMs learn language by predicting words.
• By turning videos into 3D spacetime patches and processing them with massive Transformers, these models learn emergent properties like object permanence and 3D geometry.
• They represent a shift from manually coding physics engines to letting AI learn intuitive physics purely through unsupervised observation.
• While highly capable, they still suffer from "dream physics"—statistical hallucinations where the laws of physics suddenly break down.

What to Read Next

• World Models
• Video Generation