Paper breakdown

Diffusion Transformers (DiT)

The 2022 paper that proved Transformers could replace the U-Net as the backbone for diffusion models, unlocking predictable scaling laws for image generation.

Paper: Scalable Diffusion Models with Transformers

Authors: William Peebles, Saining Xie · 2022

Read the paper

DiT operates by treating latent image patches as a sequence of tokens, feeding them through standard Transformer blocks conditioned on time and text.

The Problem

Since DDPM in 2020, every major diffusion model (including Stable Diffusion) used a Convolutional U-Net as its neural backbone. While U-Nets worked well, they were complex, required careful tuning of downsampling/upsampling blocks, and didn't exhibit the clean, predictable "scaling laws" (more compute = better performance) that Transformers demonstrated in the language domain.

The Idea

The authors proposed the Diffusion Transformer (DiT). Following the exact playbook of the Vision Transformer (ViT), they completely discarded the U-Net. They patched the latent image into a sequence of tokens and fed it through a standard Transformer architecture. They proved that diffusion models governed by Transformers exhibit excellent scaling properties: lower training loss directly correlates with better Fréchet Inception Distance (FID) scores.

How It Works

DiT works within the Latent Diffusion framework, replacing only the U-Net:

Patchify: The spatial latent representation (e.g., 32x32x4) from a VAE is chopped into patches (e.g., 2x2). These are flattened and linearly projected into a sequence of tokens.
Conditioning: Diffusion models need to know the current timestep $t$ and the class label/text $c$ . DiT injects this information using "Adaptive Layer Normalization" (adaLN-Zero). Before every Transformer block, the time and class embeddings are passed through an MLP to generate scale and shift parameters that modulate the layer normalization.
Transformer Blocks: The sequence passes through $N$ standard Transformer blocks (Self-Attention + MLP).
Unpatchify: The final sequence of tokens is reshaped back into the spatial layout of the original noise tensor.

Why It Mattered

DiT marked the convergence of language and vision architectures. It proved that the specific inductive biases of U-Nets (convolutions, spatial downsampling) were unnecessary for generation. More importantly, it showed that you could predictably improve image quality simply by increasing the depth and width of the Transformer and feeding it more data, paving the way for massive scale.

What Came After

DiT quickly replaced the U-Net as the industry standard. It is the architectural backbone of OpenAI's Sora (video generation), Stable Diffusion 3, and PixArt- $\alpha$ . The adaLN-Zero conditioning mechanism became the standard way to inject time and text into diffusion transformers.