Video Generation
A video model cleans up every frame of a clip at the same time, tying each patch to itself across frames, so the cat in frame 1 is still the same cat in frame 16.
Why Does This Exist?
You can already paint any single frame you can describe. Ask an image model for a tabby cat asleep on a blue rug sixteen times and you get sixteen lovely pictures, and sixteen different cats on sixteen different rugs. Played back at speed, that is not a film. It is a flicker.
The failure has a name: temporal inconsistency. Each frame is sampled on its own, with no memory of its neighbours, so identity, clothing, and furniture drift from frame to frame. A video model removes that drift at the root. It denoises the whole clip as one object, so frame 7 has to agree with frames 6 and 8 while it is still forming.
The one idea you need first: how still images get made. A diffusion model starts from static and cleans it step by step until a picture appears, and the full round trip is forward and reverse diffusion. This page keeps that machinery and adds the one thing it lacks: a time axis. How the clip gets squeezed small enough to work with lives in latent diffusion, and how your words steer it lives with classifier-free guidance.
Think of It Like This
A flipbook traced through thin paper
You want a flipbook of a bouncing ball, and your paper is thin enough to see the last page through the next one. So you never draw from scratch. You lay each fresh sheet over the page before it and trace the ball, shifted a little along its arc. Every page inherits its shape from its neighbour, which is exactly why the ball cannot suddenly turn square on page 9.
A video model does the tracing with arithmetic instead of paper. Each patch in each frame is blended with the same patch in the frames around it, so the shape in frame 6 constrains the shape in frame 7 the way your tracing constrains your next drawing.
Where the analogy stops: your tracing guarantees the ball stays one ball, because your hand and eye know what a ball is. The model has no hand and no ball, only pixel statistics from training clips. It holds shapes steady the way tracing does, but it understands nothing, which is why occlusions and fast motion still break it.
How It Actually Works
Three moves, all inside one network: hold the clip as one tensor, paint each frame while stitching across time, and do the heavy work compressed. Then a worked example ties the three together.
One clip is one tensor
A still image is a grid pixels tall and pixels wide with colour channels, so numbers. A clip adds frames: numbers, denoised together. In plain words: counts frames, counts colour channels, and and measure each frame in pixels.
Take a tiny toy clip with frames, channels, and frames. It holds numbers. Check it: frame-channels, pixels each, . Real clips are millions of times larger, which is why the compression step below is not optional.
Spatial layers paint, temporal layers stitch
Each block of the network, most often a video diffusion transformer, runs two attentions in turn. Spatial attention lets patches within one frame trade notes, which is what makes each single frame look right. Temporal attention then fixes one patch position and mixes only across frames: the patch on the cat's ear in frame 6 listens to the cat's ear in frames 5 and 7.
Splitting the job this way is what makes video training affordable. Full attention over every patch of every frame at once grows fast with clip length, so nearly every model factorizes it into space-then-time. The split also lets image knowledge transfer: spatial layers can start from a pretrained image model while only the temporal layers learn motion from video.
Temporal weights behave like all attention weights. For one patch they are nonnegative numbers that sum to 1, so the output is a weighted blend of what the neighbouring frames hold.
Worked example: one patch across four frames
Follow a single patch, the one on the cat's ear, across frames. Noise has left it holding the values , , , and . Left alone, frame 2 would flash bright while its neighbours sit dark. That is flicker, drawn small enough to see.
The temporal layer gives this patch the weights , , , and . They sum to , so the blend is a fair average tilted toward frame 3:
Every frame's copy of this patch now leans on the same blend, so all four converge toward instead of flashing on their own. The diagram above draws exactly this: four values in, one steady value out. Repeat that for every patch position and the clip stops shimmering.
Doing it compressed, staying on prompt
No GPU holds a long high-resolution clip as raw pixels through dozens of denoising steps. So the clip is squeezed first by a video autoencoder and denoised as a small latent, then decoded once at the end. Our toy clip's numbers squeeze to a latent of numbers, a factor of smaller. The full treatment is latent diffusion.
Two pieces finish the picture. Your words reach the clip through cross-attention to the text prompt, usually strengthened with classifier-free guidance, and many systems train the denoising path itself with flow matching. Long videos come in chunks: generate a few seconds, condition the next chunk on the last frames, repeat. That works, and it drifts, because each chunk inherits the last chunk's small mistakes along with its scene.
The trade is direct: joint denoising buys steadiness and charges memory and compute that grow with every extra frame. So do not reach for video when a still would do, when you need exact control of motion, or when correctness matters more than plausibility. The model assumes its training clips taught it how things move. Show it rare motion, heavy occlusion, or text rendered as pixels, and that assumption fails in public.
Code
This snippet is the worked example, runnable. It blends the ear patch with the four weights and checks the tensor arithmetic behind the toy clip.
import numpy as np
def temporal_blend(values: np.ndarray, weights: np.ndarray) -> float: """Blend one patch across frames; weights must sum to 1.""" return float(weights @ values)
values = np.array([0.2, 0.8, 0.6, 0.4]) # the ear patch across 4 framesweights = np.array([0.1, 0.2, 0.5, 0.2]) # temporal attention weightsprint(round(temporal_blend(values, weights), 2)) # -> 0.56print(float(weights.sum())) # -> 1.0
clip = np.zeros((4, 3, 8, 8)) # F x C x H x W toy cliplatent = np.zeros((4, 4, 2, 2)) # squeezed autoencoder latentprint(clip.size, latent.size, clip.size // latent.size) # -> 768 64 12The numbers match the worked example and the diagram: the weights sum to 1, the patch settles at 0.56, and squeezing 768 values down to 64 cuts the load twelvefold.
Watch Out For
The cast changes halfway through the scene
The first frames look perfect, then a hand regrows its fingers, a logo reflows, or the tabby turns long-haired after walking behind a chair. The longer the clip and the faster the motion, the sooner it happens.
The model never promised you an object, only plausible pixels, so anything hidden and revealed again gets reinvented. Keep shots short, keep motion small, and pin the look down with an input image or a strong prompt when the tool allows it. Check hands, text, and repeating patterns before you trust a clip, and never use one where continuity carries meaning, like instructions or evidence.
Seams and strobing in long clips
Every few seconds the scene jumps, or a fine texture like grass, fabric, or water pulses at a steady beat. The jumps sit exactly on chunk boundaries. The pulse comes from frames disagreeing about detail the autoencoder blurred.
Both are the price of building long videos from short pieces. Overlap the chunks and condition each new piece on the last frames with a fixed seed instead of starting fresh, and generate the longest single chunk your memory allows so there are fewer seams to hide. If a texture strobes, simplify it in the prompt: the model cannot hold detail its own compressor threw away.
The Quick Version
- Video generation denoises a whole clip as one tensor instead of sampling frames one by one.
- Temporal attention blends each patch with itself across neighbouring frames, which is what stops the flicker.
- Weights of , , , and turn patch values of , , , and into one steady .
- A video autoencoder squeezes the clip to a small latent for denoising, then decodes once. The toy clip drops from 768 values to 64.
- Joint denoising costs memory and compute per frame, so stills, exact motion control, and factual tasks belong to other tools.
- Fakes look real, so knowing the detector matters too. How they get caught lives in synthetic media detection.