Imagen

The Problem

Prior text-to-image models like DALL-E 2 relied on paired image-text data to train specialized encoders (like CLIP). But natural language is vastly richer and more complex than image captions. Training a text encoder exclusively on image captions limits its ability to understand complex compositionality, long-form descriptions, and nuanced spatial relationships. Text-to-image models were fundamentally constrained by the language understanding of their text encoders.

The Idea

Instead of training a text encoder from scratch on image-text pairs, what if we just drop in a massive, off-the-shelf Large Language Model (like T5-XXL) that was already trained on a huge text-only corpus? The core insight of Imagen is that scaling the size of a pre-trained, frozen language model improves image generation quality and text alignment far more effectively than scaling the size of the image diffusion model itself.

How It Works

Frozen Language Model Imagen uses a frozen T5-XXL encoder (trained on text only) to convert the user's prompt into a sequence of rich text embeddings. Because it's frozen, the model retains its deep understanding of syntax and semantics without unlearning it during image training.

Cascaded Diffusion Architecture The text embeddings are fed into a series of three diffusion models:

1. Base Model: A text-to-image diffusion model that generates a small 64×64 pixel image.
2. Super-Resolution Model 1: Upsamples the 64×64 image to 256×256.
3. Super-Resolution Model 2: Upsamples the 256×256 image to 1024×1024.

Efficient U-Net The base 64×64 model introduces structural modifications to the standard U-Net, optimizing it for speed and memory efficiency while improving convergence at low resolutions.

Dynamic Thresholding High classifier-free guidance weights normally cause color saturation and artifacts. Imagen introduces Dynamic Thresholding: at each sampling step, it clips the pixel values based on a certain percentile of the pixel distribution rather than a static bound. This allows Imagen to use very high guidance weights (which ensure strong text alignment) without burning out the image colors.

Why It Mattered

Imagen proved that language models are excellent zero-shot text encoders for image generation. It achieved state-of-the-art results on COCO without ever training on COCO, and it demonstrated unprecedented text alignment—accurately rendering text inside images (like signs or shirts) and handling complex spatial prompts better than models using CLIP.

What Came After

Imagen's success heavily influenced the next generation of text-to-image models. The idea of using powerful, frozen language models (often T5) instead of or alongside CLIP became a standard architectural choice, seen in later models like eDiff-I and Stable Diffusion 3. Google later productized the architecture into Imagen 2 and Imagen 3.