Skip to content
AI360XpertBeta

Diffusion Language Models

Instead of writing left to right one word at a time, a diffusion language model starts from a line of blanks and fills the whole line in at once, refining every word together over a few fixed passes.

Starting from four masked slots, a diffusion language model fills the whole line in two parallel refinement steps instead of writing left to right
Starting from four masked slots, a diffusion language model fills the whole line in two parallel refinement steps instead of writing left to right

Why Does This Exist?

How LLMs work describes the standard deal: a model writes left to right, committing to one token before it sees the next. That works, but it carries two costs you feel in practice. First, positions can't share work. Token 100 waits on tokens 1 through 99, so generation is stubbornly serial. Second, there's no revision. Pick a bad word in the opening clause and the model must live with it, talking its way around the damage for the rest of the passage. The rules for picking each token are covered in decoding strategies. They tune the choice, but they don't remove the serial bottleneck.

Diffusion language models drop the left-to-right order. They fix the length up front, start from a full line of blanks, and refine every position together over a fixed number of passes. Because each pass sees the whole line, a word on the left can be chosen to fit a word on the right. That global planning is the payoff, and the idea comes straight from image diffusion: diffusion models build a picture by denoising all pixels at once, and forward and reverse diffusion explains the noise schedule this page leans on. Here you'll learn the masked formulation of that idea for text: the setup, one inference pass and one training step with real numbers, and what the parallelism costs.

The one idea you need first. Next-token prediction: a standard model reads the tokens so far, x1x_1 through xt−1x_{t-1} (the sequence indexed by position), and outputs a probability distribution over the vocabulary for the next slot. A [MASK] token is a placeholder meaning "undecided", borrowed from BERT, which trained by guessing hidden words exactly like that. If that sentence parses, you're ready.

Think of It Like This

A test paper with every answer blanked out

Picture a printed quiz where someone whited out every answer, four blanks per line. You don't answer question 1 in ink and move on. You pencil a guess into every blank at once. Then you scan the page, keep the answers you're most confident in, erase the shaky ones back to blank, and guess again with the kept answers as context. After a few passes, you ink everything in.

That maps onto the mechanism point for point. Blanks are masked slots, pencil guesses are parallel predictions, erasing the shakiest is confidence-based remasking, and ink is the final decode. Where it stops: you can ask the teacher for extra blanks mid-quiz, but the model can't. Its length is fixed before pass one, and every pass re-reads the entire line, so a four-word answer costs the same number of passes as a forty-word one.

How It Actually Works

There are two ways to diffuse text. The older one embeds words into continuous vectors, adds Gaussian noise there, and rounds back to words at the very end. The one this page teaches never leaves word-land: it masks words and learns to unmask them. The math is cleaner, training is close kin to BERT with one big upgrade (BERT fills blanks to understand; a masked diffusion model iterates the fill until it becomes generation), and every number below is hand-computed so you can check it.

The setup

Fix a length LL (the number of token slots; our running example uses L=4L = 4) and write the clean sentence as x0x_0 (subscript 0 means zero noise). The noisy version at noise level tt is xtx_t, where tt runs from 0 (clean) to 1 (fully masked). The forward process builds xtx_t from x0x_0 by replacing each token with [MASK] independently with probability 1−αt1 - \alpha_t. Here αt\alpha_t (alpha sub tt) is the keep probability: the expected fraction of tokens left visible. At t=0t = 0, α0=1\alpha_0 = 1 and nothing is masked; at t=1t = 1, α1=0\alpha_1 = 0 and every slot is blank. The symbol θ\theta (theta) denotes the model's weights, so pθp_\theta is the probability distribution the model outputs.

Inference runs the other way. Start at x1x_1: all blanks. Ask the model to predict every blank at once, keep the most confident predictions, re-blank the rest, and repeat for a fixed SS steps. Our example uses S=2S = 2 on the sentence "the cat sat down", exactly as drawn above: the schedule keeps 1 token on pass one and everything remaining on pass two.

One inference pass, worked

Pass one sees [MASK] [MASK] [MASK] [MASK] and scores all four slots. Suppose it returns 0.300.30 for "the", 0.900.90 for "cat", 0.250.25 for "sat" and 0.200.20 for "down" at their slots. You keep "cat" and re-blank the rest: [MASK] [cat] [MASK] [MASK]. Pass two predicts with "cat" as context and returns 0.850.85 for "the", 0.800.80 for "sat" and 0.750.75 for "down". The schedule keeps all three, so the line reads "the cat sat down". Two passes, four words, and "the" was chosen with "cat" already visible to its right. A left-to-right model never gets that: it picks "the" knowing only what came before.

One training step, worked

Training never runs the full chain. It samples a noise level, masks to match, and asks the model to recover the originals. Say t=0.5t = 0.5 with a linear schedule, so each token is masked with probability 0.50.5, and the draw gives [the] [MASK] [MASK] [down]. The model assigns probability 0.700.70 to the true word "cat" and 0.500.50 to "sat". Cross-entropy for one slot is −log⁡p-\log p (the natural log of the model's probability on the right answer, negated), so:

−log⁡(0.70)≈0.3567,−log⁡(0.50)≈0.6931-\log(0.70) \approx 0.3567, \quad -\log(0.50) \approx 0.6931

Average over the two masked slots: (0.3567+0.6931)/2≈0.5249(0.3567 + 0.6931) / 2 \approx 0.5249. Steps with heavy masking contribute many such terms and steps with light masking contribute few, so the loss re-weights each step by 1/t1/t (one over the noise level) to keep every noise level pulling its weight. Here 1/0.5=21/0.5 = 2, giving 2×0.5249≈1.04982 \times 0.5249 \approx 1.0498. The snippet below recomputes all four numbers.

What the parallelism costs

Three bills come due. First, length: LL is locked before pass one. Guess short and the thought gets cut off; guess long and every pass re-reads padding you didn't need. Second, steps: SS doesn't care that your answer is four words. Short replies cost the same passes as long ones, so for quick chatty text the serial model wins on latency. Third, memory tricks don't transfer. A standard model's cache reuses past work because the past never changes, but here the whole line changes every pass, so each pass is a full read.

So use diffusion LMs where global shape matters: filling a gap mid-sentence, forcing a fixed ending, or revising a draft as a whole. Don't use them for streaming short answers token by token. The continuous variant runs the same loop in embedding space (noise vectors instead of blanks, one rounding step to words at the end). Same trade-offs, plus rounding error where a vector lands between two words.

Code

This snippet recomputes the training numbers above, so you can check the arithmetic instead of trusting it.

import math

def cross_entropy(p_true: float) -> float:    """Negative log-likelihood of the correct word (natural log)."""    return -math.log(p_true)

def masked_diffusion_loss(probs: list[float], t: float) -> float:    """Mean cross-entropy over masked slots, re-weighted by 1/t."""    mean_ce = sum(cross_entropy(p) for p in probs) / len(probs)    return mean_ce / t

print(round(cross_entropy(0.70), 4))  # -> 0.3567print(round(cross_entropy(0.50), 4))  # -> 0.6931print(round((cross_entropy(0.70) + cross_entropy(0.50)) / 2, 4))  # -> 0.5249print(round(masked_diffusion_loss([0.70, 0.50], 0.5), 4))  # -> 1.0498

All four match the hand computation: the two per-slot terms, their mean, and the re-weighted loss.

Watch Out For

Locking the length before you know it

The symptom is answers that trail off mid-thought or drown in filler, and lengthening LL makes everything slower because every pass re-reads the padding. The cause is treating LL like a guess instead of a budget: it caps content and sets cost at the same time. Size LL from your own data (measure the lengths of real answers and cover nearly all of them, don't eyeball it), and build long pieces as chained fixed-length blocks rather than one giant line.

Stopping after one pass

The symptom is fluent-looking garbage. A single parallel prediction is just fill-in-the-blank: every slot is guessed without settled neighbours, so errors agree with each other instead of cancelling out. If your samples look far worse than the model's masked predictions suggested, you're under-iterating. Run the full SS-step schedule and unmask high-confidence slots first. The quality comes from the iteration, not from one giant parallel guess.

The Quick Version

  • A diffusion language model starts from a fully masked line and refines every slot together over a fixed number of passes.
  • Each pass predicts all masked slots at once, keeps the most confident, and re-blanks the rest until nothing is masked.
  • Training samples a noise level, masks to match, and averages cross-entropy over masked slots, re-weighted by 1/t.
  • Because every pass sees the whole line, left words can fit right words, which makes infilling and constrained endings natural.
  • Length and step count are fixed before pass one, so short answers cost as much as long ones and streaming doesn't apply.