VideoPoet: A Large Language Model for Zero-Shot Video Generation

The Problem

Most early video generators relied on diffusion models. While diffusion produces high-quality continuous pixels, it requires complex architectural hacks (like interleaving spatial and temporal layers) to model long-term consistency. Meanwhile, Large Language Models (LLMs) like GPT-4 had proven that autoregressive Transformers were unmatched at maintaining long-context consistency—but they operate on discrete text tokens, not continuous video pixels.

The Idea

VideoPoet proposed that video generation shouldn't use diffusion at all. Instead, it should be treated exactly like text generation. By compressing video frames and audio clips into discrete "tokens" (just like words in a sentence), VideoPoet could use a standard, off-the-shelf autoregressive LLM architecture to generate video and audio simultaneously.

How It Works

1. Unified Tokenization The secret sauce of VideoPoet is converting continuous data into discrete tokens:

• Video: Uses MAGVIT-v2, a highly advanced 3D tokenizer that compresses chunks of video frames into a grid of discrete integers.
• Audio: Uses SoundStream to compress audio waveforms into discrete audio tokens.
• Text: Uses standard subword tokenization.

2. The Unified Sequence A training example is flattened into a single 1D sequence of tokens: [Text Tokens] -> [Video Tokens] -> [Audio Tokens]

3. Autoregressive Generation A massive decoder-only Transformer is trained on these sequences using the exact same objective as ChatGPT: next-token prediction. Given the text tokens, predict the first video token. Given the text and first video token, predict the second video token, and so on. Because it's a standard LLM, it inherits the Transformer's ability to maintain long-term coherence, ensuring the video doesn't morph randomly over time.

4. Multi-Task Capabilities Because the model learns the joint distribution of text, video, and audio, it can perform multiple tasks zero-shot depending on what input is provided:

• Text-to-Video (Input: Text, Output: Video)
• Video-to-Audio (Input: Video, Output: Audio)
• Video Outpainting/Extension (Input: Video chunk, Output: Next Video chunk)

Why It Mattered

VideoPoet proved that the LLM paradigm (discrete tokenization + autoregressive Transformers) was highly competitive with, and in some aspects superior to, diffusion for video generation. It produced highly dynamic motion, perfectly matched audio, and maintained temporal consistency over long sequences, challenging the dominance of diffusion in the generative video space.

What Came After

VideoPoet heavily influenced the development of multimodal "native" LLMs, paving the way for models like Gemini 1.5 and later iterations of video models that merge the LLM and diffusion paradigms.