Gen AI

Video Generation

Generating a video isn't just about generating 24 images per second; it's about forcing the AI to remember what the previous images looked like so the cat doesn't turn into a dog halfway through the clip.

Why Does This Exist?

When Text-to-Image models matured, the immediate next question was: "Can we make a movie?" At a basic level, a video is just a sequence of images (frames) played very quickly, usually 24 to 60 frames per second. Naively, you could just ask a standard image generator to create 24 images of "a man walking," and stitch them together.

The problem is Temporal Inconsistency. The image model doesn't know that frame 2 is supposed to be the direct continuation of frame 1. It will generate a man walking in a blue shirt in frame 1, a red shirt in frame 2, and maybe a woman walking in frame 3. The resulting video will be a violently flickering, hallucinatory mess. Video Generation models exist to solve this exact problem: they explicitly model the dimension of time, ensuring physics, geometry, and identity remain perfectly stable from the first frame to the last.

Think of It Like This

The Flipbook Artists

Imagine hiring 100 artists to draw a 100-page flipbook of a bouncing ball.

Naive Image Generation: You put all 100 artists in separate, soundproof rooms. You tell them all to draw a bouncing ball. When you staple their pages together, the ball changes size, color, and location randomly on every page.
Video Generation: You put the 100 artists at a long table. You enforce a rule: before you draw your page, you must look at the page of the artist to your left (the past) and the artist to your right (the future). Because they are forced to pay attention to each other, the resulting flipbook is a perfectly smooth animation of a single bouncing ball.

How It Actually Works

Creating a video generation model usually involves taking an architecture that is already good at images (like a U-Net or a Diffusion Transformer) and upgrading it to handle the dimension of time.

1. The Temporal Dimension

An image is a 3D tensor: Channels $\times$ Height $\times$ Width (C $\times$ H $\times$ W). A video is a 4D tensor: Frames $\times$ Channels $\times$ Height $\times$ Width (F $\times$ C $\times$ H $\times$ W). When a noisy video is fed into a diffusion model, the network has to denoise all the frames simultaneously.

2. Temporal Attention

The magic happens by adding "Temporal Layers" to the neural network. In a standard image model, there are Spatial Attention layers: a pixel in the top-left corner looks at a pixel in the bottom-right corner to make sure the image makes sense spatially. In a video model, we insert Temporal Attention layers right after the spatial ones. In a temporal attention layer, a pixel at coordinate (X, Y) in Frame 1 looks only at coordinate (X, Y) in Frame 2, Frame 3, etc. This forces the model to learn how objects move through time, entirely separately from how they look in a single frame.

3. Joint Training (Images and Video)

A massive breakthrough (popularized by models like Sora) was joint training. Video data is incredibly expensive and rare compared to image data. Researchers realized they could train the model on a mix of images (F=1) and videos (F=24, F=60). The model learns spatial beauty, lighting, and texture from the billions of static images, and it learns physics and motion from the smaller set of videos.

4. Autoregressive and Latent Scaling

Generating 60 frames of high-resolution video simultaneously requires an astronomical amount of VRAM—far more than fits on a single GPU. To solve this, modern models heavily compress the video using a 3D Autoencoder (Latent Diffusion). Furthermore, to generate a 10-minute video, they often generate the first 5 seconds, use the last frame as the "conditioning" for the next 5 seconds, and generate it autoregressively in chunks.

Show Me the Code

This pseudocode shows how spatial and temporal attention are interleaved inside a video generation block.

1import torch2import torch.nn as nn3
4class VideoTransformerBlock(nn.Module):5    def __init__(self, embed_dim):6        super().__init__()7        # Spatial attention: Looks across Height and Width8        self.spatial_attention = nn.MultiheadAttention(embed_dim, num_heads=8)9        10        # Temporal attention: Looks across Frames11        self.temporal_attention = nn.MultiheadAttention(embed_dim, num_heads=8)12        13        self.feed_forward = nn.Linear(embed_dim, embed_dim)14
15    def forward(self, x):16        # x is a flattened sequence of patches: (Batch, Frames, Patches_per_frame, Embed_Dim)17        B, F, P, D = x.shape18        19        # 1. Spatial Attention20        # We merge Batch and Frames to treat every frame independently21        x_spatial = x.view(B * F, P, D)22        # Each patch looks at other patches in the SAME frame23        x_spatial, _ = self.spatial_attention(x_spatial, x_spatial, x_spatial)24        x = x_spatial.view(B, F, P, D)25        26        # 2. Temporal Attention27        # We transpose to merge Batch and Patches28        x_temporal = x.transpose(1, 2).reshape(B * P, F, D)29        # Each patch looks at the exact SAME patch across DIFFERENT frames30        x_temporal, _ = self.temporal_attention(x_temporal, x_temporal, x_temporal)31        32        # Reshape back to original33        x = x_temporal.view(B, P, F, D).transpose(1, 2)34        35        # 3. Feed Forward36        x = self.feed_forward(x)37        38        return x

Watch Out For

Physics Hallucinations

Because video models learn physics strictly by watching 2D pixels move over time, they do not actually understand 3D geometry or mass. They are approximating physics. This is why AI videos often feature surreal errors: someone biting into a cookie but the cookie remains whole, or a person walking in front of a chair but their leg passes directly through the chair. The model doesn't know what solid matter is; it just knows what pixels usually do.

The Quick Version

Generating a video frame-by-frame using an image model results in severe temporal inconsistency (flickering and morphing).
Video models solve this by treating a video as a 4D tensor and denoising all frames simultaneously.
They achieve this by interleaving Spatial Attention (understanding the image) with Temporal Attention (understanding how pixels move across frames).
Modern video models are trained jointly on massive datasets of static images (for visual quality) and videos (for motion and physics).
Because 4D tensors are massive, video generation heavily relies on latent compression (DiT and 3D VAEs) to fit in GPU memory.