U-Net vs. Diffusion Transformers (DiT)
A diffusion model needs a denoiser that takes a noisy image and guesses the noise. U-Net does that with shrinking convolutions, while a Diffusion Transformer cuts the image into patches and lets every patch talk to every other patch.
Why Does This Exist?
A diffusion model is only as good as its denoiser. At each step you hand the network a noisy image and ask it to guess the noise so you can subtract it. That job belongs to one backbone network, run dozens of times per image. Pick the wrong backbone and you either run out of memory or hit a ceiling where extra compute stops helping.
The first backbone that worked was the U-Net. It came from medical segmentation, where the task is also image in and image out at the same size. It shrinks the image with convolutions to grasp the scene, then rebuilds it while reusing fine edges through skip connections. If you want the full denoising loop around it, start with diffusion models and forward and reverse diffusion.
The problem showed up when teams tried to scale. Convolutions look at small neighborhoods, so long range structure has to propagate through many layers. Widening and deepening a U-Net helped, but the gains were uneven and fiddly to tune. Meanwhile transformers were scaling cleanly in language: more parameters plus more compute gave better loss with few surprises. The Diffusion Transformer, usually shortened to DiT, asks whether you can treat an image like a sentence and get the same scaling deal.
This page compares the two backbones head to head. It doesn't re-teach the noise schedule or the sampler. For the schedule, read DDPM. For what happens after the backbone guesses the noise, read diffusion samplers. For why both backbones now run on compressed images instead of pixels, read latent diffusion.
The one idea you need first: a convolution mixes a pixel with its close neighbors using a small learned kernel. A 3 by 3 kernel sees 9 pixels at a time. That gives a U-Net a strong local bias: edges and textures come cheap.
Self-attention does the opposite. Every token compares itself with every other token in one step, so one layer can link an eye patch to a mouth patch across the image. The price is pairwise work. The full mechanism is in self-attention, and the vision habit of cutting images into patches is in vision transformers.
Think of It Like This
Two ways to restore a tiled wall
You have a bathroom wall covered in small tiles, and years of grime hide the pattern. You hire two restorers.
The first restorer photographs the wall, prints a tiny thumbnail, and studies the thumbnail to work out the big pattern. Then she repaints the wall at full size while an assistant holds the original photo beside her hand so no edge drifts. The thumbnail is the bottleneck. The assistant holding the photo is the skip connection. She never loses the fine grout lines because she kept a copy of them.
The second restorer photocopies the wall, cuts the copy into four squares, and lays the squares on a table. She compares every square with every other square before cleaning any of them, so a corner of a flower in square one tells her what the smudge in square four should be. She never shrinks anything. The comparison across squares is the attention step.
Where it stops: tiles don't move, and your wall has no timestep, no text prompt, and no learned weights. A real denoiser is told how noisy the input is and what the image should show, and it learns its cleaning rule from data. The analogy covers only the geometry habit: shrink and rebuild with a saved copy, versus cut up and compare everything at once.
How It Actually Works
Both backbones solve the same regression. Let be the noisy image at timestep , where is an integer naming how much noise was added. Let be the true noise that was added. Let be the network prediction of that noise, where means the learned weights. The network reads plus plus an optional text embedding, and it outputs with the same height and width as . The sampler then uses that guess to step toward a cleaner image.
The U-Net path: shrink, then rebuild with help
A U-Net keeps the image as a grid the whole way. The down path applies convolutions and halves the height and width once or twice per stage while adding channels. Halving twice turns a 4 by 4 map into a 2 by 2 map and then a 1 by 1 summary, which forces the network to decide what the scene shows. The up path doubles the size back in mirrors of those steps.
The part you must not drop is the skip. Before each halving, the network saves a copy of the high resolution map and concatenates it, meaning stacks it channelwise, with the matching up path map later. Without that copy, thin edges and text would have to survive the tiny bottleneck, and they don't. With it, the bottleneck carries what the image means and the skip carries where the edges sit.
Cost grows gently with size because a convolution touches each pixel a fixed number of times. That locality is also the limit. One 3 by 3 layer cannot connect distant patches, so global layout has to emerge across depth.
The DiT path: cut into patches, then let patches vote
A DiT throws away the grid early. Let be image height and be image width in latents, where a latent is the compressed pixel grid from latent diffusion. Let be patch size in latents per side. Let be token count, the number of patches. Then:
In plain words: divide each side by the patch size, then multiply. Each patch is flattened, meaning its small square of pixels is unrolled into one vector, then linearly projected into a token , where runs from 1 to and each token lives in dimensions. Add a positional embedding so token 1 still knows it came from the top left, add the timestep and text embeddings, and run plain transformer blocks. Finally reshape, meaning fold the sequence back into a grid, to output .
There's no shrinking and no skip in the classic layout. Global structure comes free because self-attention compares all tokens in one layer. The bill is pairwise: the attention matrix holds scores, so doubling image width quadruples and multiplies attention work by 16. You'll feel that in the pitfall below.
Worked example: one 4 by 4 latent, 2 by 2 patches
Take , , .
Token count:
So you get 4 tokens, named through . Each patch holds latent values, so flattening gives a length 4 vector per token before projection. Total values: , which matches the latents you started with. Nothing lost, only rearranged.
Attention scores:
Sixteen pairwise comparisons link every patch to every patch, including each patch to itself. That 4 by 4 score grid is the small square in the diagram.
Now double each side to , with the same :
Tokens grew from 4 to 16, a factor of 4. Attention entries grew from 16 to 256, a factor of 16. That squaring is why DiTs in practice run on small latents, not full resolution pixels.
The same 4 by 4 input through a tiny U-Net would read: 4 by 4 in, down to 2 by 2, back to 4 by 4 out, with the 4 by 4 input copied around the bottleneck through the skip. Same in size, same out size, and the skip is what keeps the rebuild sharp.
Code
The point of this snippet is counting, because the counting is where people get surprised. It computes token counts and attention entries for the two sizes in the worked example.
def patch_count(h: int, w: int, p: int) -> int: return (h // p) * (w // p)
def attention_scores(n: int) -> int: return n * n
n_small = patch_count(4, 4, 2)n_large = patch_count(8, 8, 2)print(n_small) # -> 4print(attention_scores(n_small)) # -> 16print(n_large) # -> 16print(attention_scores(n_large)) # -> 256Four tokens give 16 scores. Sixteen tokens give 256 scores. Doubling each side quadruples the sequence and multiplies attention work by 16. When you size a DiT run, this is the arithmetic to do first.
If you steer either backbone with extra inputs, the wiring differs but the idea matches: U-Nets take spatial maps through adapters, DiTs take extra tokens. The spatial adapter pattern is in ControlNet and spatial conditioning, and the prompt weighting trick both use is in classifier-free guidance.
Watch Out For
Doubling resolution blows up DiT attention 16x
Symptom: training fits at low resolution, then a doubled resolution dies with out of memory or crawls, even though the U-Net baseline at the same resolution was fine. You quadrupled the tokens, so attention entries went up 16x, exactly the 16 to 256 jump in the example above.
Fix: keep DiT on latents per latent diffusion, never raw pixels, and change one thing at a time. Raise from 2 to 4 to cut by 4, or shorten the sequence with windowed attention, or grow resolution in stages. Do the and math before you launch.
Patches without positions scramble the image
Symptom: the sampler runs, loss looks plausible, but outputs have eyes, wheels, or windows in impossible places, or texture that ignores the prompt layout. Someone removed or broke the positional embeddings, so token 1 and token 4 look interchangeable to the transformer.
Fix: check that each carries its grid position and that timestep and the text embedding actually reach every block. The U-Net mirror of this bug is deleting skips: outputs turn mushy because fine positions never bypass the bottleneck. If layout is wrong, check positions and skips before you retrain anything.
The Quick Version
- A diffusion backbone takes a noisy image plus timestep and predicts the noise at the same size.
- A U-Net shrinks the grid with convolutions and rebuilds it, reusing saved high resolution maps through skip connections to keep edges sharp.
- A Diffusion Transformer cuts the image into patch tokens and runs self-attention, so every patch sees every other patch in one layer.
- A 4 by 4 latent with 2 by 2 patches gives 4 tokens and 16 attention scores, while 8 by 8 gives 16 tokens and 256 scores.
- DiTs trade quadratic attention cost for cleaner scaling and global context, while U-Nets trade global reach for cheaper high resolution detail.
- Use a U-Net when compute is tight or data is small, and a DiT on latents when you plan to scale parameters and data together.