GLIDE

The Problem

Before GLIDE, diffusion models had proven they could synthesize incredibly high-fidelity images, but guiding them precisely with text remained a challenge. Prior approaches like CLIP-guided diffusion required computing gradients through an external image-text classifier (like CLIP) at every sampling step. This added significant computational overhead, made the sampling process unstable due to adversarial gradients, and often compromised the natural photorealism of the output. The bottleneck was relying on a separate, noisy classifier to steer a powerful unconditional generative model.

The Idea

The authors realized that instead of using an external classifier, they could train a single diffusion model that internally understood text and guide it using classifier-free guidance. By training a text-conditional diffusion model and occasionally dropping the text prompt during training (replacing it with an empty sequence), the model learns both a conditional and an unconditional distribution. During inference, they extrapolate away from the unconditional prediction towards the text-conditional prediction, effectively amplifying the text's influence without ever computing a classifier gradient.

How It Works

Text Encoding GLIDE begins by encoding the text prompt into a sequence of token representations. These embeddings are fed into the diffusion model's U-Net architecture. Specifically, the model uses a Transformer to process the text, providing both a global text representation (added to the diffusion timestep embedding) and local sequence features (injected via cross-attention into the spatial layers of the U-Net).

Classifier-Free Guidance During training, the text embedding is replaced with a null embedding 20% of the time. This forces the model to learn $p(x | \text{text})$ and $p(x | \emptyset)$. At generation time, the model predicts the noise twice: once with the text prompt, and once with the null prompt. The final prediction is a linear combination that pushes the output away from the unconditional image and heavily towards the text-guided image, governed by a guidance scale parameter.

Image Inpainting and Editing Because GLIDE inherently understands both image spatial structures and text semantics, it can be seamlessly used for editing. The model was explicitly fine-tuned for inpainting by masking out random regions of the image during training and providing the unmasked regions alongside the mask as additional conditioning channels. This allowed users to select an area of an image and use text to insert new objects or alter the scene with perfect lighting and shadow matching.

Why It Mattered

GLIDE proved definitively that classifier-free guidance was superior to CLIP guidance for text-to-image synthesis. The resulting model not only produced vastly more realistic images, but it also adhered much more accurately to complex textual prompts. It eliminated the need for fragile external classifiers during inference and established the architectural blueprint for modern text-to-image models: text-conditioned U-Nets amplified by classifier-free guidance.

What Came After

GLIDE laid the direct foundation for DALL-E 2, which scaled up these principles and introduced diffusion priors. The combination of cross-attention for text injection and classifier-free guidance for prompt adherence became the standard, eventually powering open-weights models like Stable Diffusion and midjourney.