Diffusion Transformers

Diffusion Transformers swap the UNet denoiser for a plain transformer over latent patches, so image quality scales with compute like language models do.

DiT patchifies the noisy latent into tokens and denoises them with conditioned transformer blocks

Why Does This Exist?

UNet denoisers carry heavy inductive bias: fixed resolutions, hand-placed attention, convolution habits from another era. Scaling them up helps, but unevenly, and every size needs architectural babysitting. Language models showed the alternative: one uniform block repeated, with quality marching upward as parameters and data grow.

DiT ports that deal to vision. Patchify the noisy latent, add timestep and class conditioning through adaptive normalisation, and stack identical transformer blocks. The same code trains a small model and a huge one, and the huge one wins predictably. Video generators adopted the backbone directly, as video diffusion shows.

Think of It Like This

A quilting circle versus a single quilt pattern

A UNet is a quilting circle where each member owns one fixed job: borders, centre, corners. Adding members means renegotiating every role. A transformer is one quilt pattern repeated block by block: to go bigger you simply sew more identical squares, and the blanket improves uniformly.

Patch tokens are the squares and adaLN-Zero is the stitching that starts each new square as a pass-through. The circle fails when squares must differ structurally, tasks where convolution bias genuinely helps, which small-data regimes still favour.

How It Actually Works

The noisy latent $x_t$ , say $64 \times 64 \times 4$ , splits into $p \times p$ patches, linearly embedded as tokens with positional encodings. Each DiT block applies adaptive layer norm with zero initialisation (adaLN-Zero): conditioning vectors regress scale $\gamma$ and shift $\beta$ per block, initialised so every block starts as the identity function:

h' = h + \gamma_0 \cdot \mathrm{Attn}(\mathrm{Norm}(h)), \qquad \gamma_0 = 0 \text{ at start}

Timestep and class labels enter through the same conditioning path, no cross-attention needed. After the final block, a linear head unpatchifies tokens back to predicted noise or velocity.

Worked example

Patch size $2$ on a $64 \times 64$ latent gives a $32 \times 32$ token grid: $1{,}024$ tokens, each embedding a $2 \times 2 \times 4 = 16$ -number patch. Self-attention over $1{,}024$ tokens costs about a million pairwise interactions per head per block, heavy but uniform, and identical at every scale. Doubling width quadruples that cost while predictably lowering FID, the scaling curve that UNets never offered cleanly.

Code

Token counting for patchified latents, the sizing arithmetic behind DiT configs.

1def dit_tokens(latent_px: int = 64, patch: int = 2, channels: int = 4) -> tuple[int, int]:2    """Token count and numbers per patch for a square latent."""3    tokens = (latent_px // patch) ** 24    per_patch = patch * patch * channels5    return tokens, per_patch6
7
8print(dit_tokens())  # -> (1024, 16) tokens, numbers per patch

Watch Out For

Expecting UNet sample efficiency at small scale

Transformers learn spatial structure from data rather than baking it in. On small datasets DiT underperforms tuned UNets for the same compute, because the bias it discarded was doing real work there.

Use DiT where data and compute scale; keep convolutional denoisers for small-data regimes. Architecture choice follows data budget, not fashion.

Breaking adaLN-Zero initialisation

The zero-init identity start is load-bearing: early training would otherwise route large random signals through deep attention stacks. Initialising conditioning projections like ordinary layers reintroduces the instability the design removed.

Verify block outputs equal inputs at initialisation in a unit test. One nonzero gamma at step zero can destabilise the first thousand steps.

The Quick Version

DiT replaces the UNet with identical transformer blocks over patchified latent tokens.
AdaLN-Zero injects timestep and class conditioning starting from identity blocks.
A $64$ latent at patch $2$ yields $1{,}024$ tokens of $16$ numbers each.
Quality scales predictably with size, compute, and data like language models.
Small-data regimes still favour convolutions, and zero-init must be verified, not assumed.