Visual explainer
BERT vs GPT
GPT predicts continuations with causal attention; BERT reconstructs selected tokens with bidirectional context. Trace the masks and a concrete leakage test.
GPT's causal language-model objective predicts a next token from a prefix. BERT's masked language-model objective reconstructs selected input tokens using context on both sides. Those training objectives favor different uses; they do not make classification exclusive to BERT or all generation impossible for encoders.
We use the illustrative tokens the, cat, sat and later down. Treat each word as one token solely to keep the diagrams small; actual tokenizers may split words. The dashed slots contrast continuation with reconstruction. This page compares the original architectures and objectives, not capabilities of every later model carrying either name.
Predict From A Prefix
After the cat, our synthetic three-candidate distribution is sat .60, ran .25, hid .15. The bar lengths are 120, 50 and 30 SVG units at 200 units per probability. They sum to one. These are chosen teaching values, not measured GPT outputs; choosing the largest probability does not establish correctness. A real vocabulary contains many more candidates.
Read The Causal Mask
Rows are query positions and columns are key positions. A dot permits attention; a cross blocks it. Row two permits columns one and two, including its own current token, while column three is unavailable. There are six allowed pairs across the three rows. Attention is not limited to the immediately preceding token.
In a causal Transformer, the mask adds negative infinity to forbidden attention scores before softmax. Thus their attention weights are zero. The next-token targets are shifted: the representation at the predicts cat, and the one at cat predicts sat. Padding and special tokens are omitted from this miniature grid.
Reconstruct A Selected Token
Replace cat with a mask token. Both the and sat can contribute to its representation before the masked-language-model head predicts the original token. The paths show permitted information flow, not measured attention weights or guaranteed answers. The original identity is withheld in this example; the surrounding words may still leave ambiguity.
The BERT paper, section 3.1, describes selected-token prediction with a mix of masking, random replacement and unchanged selected tokens. It also uses a next-sentence prediction objective. We isolate the masked-token mechanism here rather than depicting the entire pretraining recipe.
Compare Allowed Connections
On exactly the same grid scale, the selected second row can now read all three positions. Nine pairs are allowed overall. This means the attention mask permits a connection, not that learned weights are equal or useful. BERT still needs padding masks, and input corruption is a different operation from blocking future attention. A larger visible region is not an accuracy score.
Both Can Classify
The circles are hidden states, with qualitative paths showing which sentence information can reach the selected state. A causal final state can incorporate the whole supplied prefix; a bidirectional classification-token state can incorporate the supplied sentence. A task-trained head maps the selected state to class scores. The arrows and head boxes do not compare measured quality.
The original GPT paper, section 3.2, explicitly fine-tunes a final-token representation for supervised classification. BERT section 3.2 uses its classification token for sentence-level tasks. Task data, training, resource limits and evaluation determine the choice; decoder versus encoder is not a capability wall.
Prevent A Training Shortcut
During next-token training, the supplied sequence includes the cat, but the position at the must predict cat without reading it. The amber shortcut depicts a forbidden path from that future input into the earlier representation. Without the causal mask, the training task can leak its own answer. This differs from BERT's legitimate use of future context to reconstruct a corrupted position.
Work One Attention Row
Isolate one attention head and give position two equal scores for all permitted keys. Choose scalar values (0, 0, 3). Causal softmax gives weights (1/2, 1/2, 0), producing 0. Full attention gives (1/3, 1/3, 1/3), producing 1. The output bar uses 100 units per scalar unit; a vertical cap denotes zero length.
These are attention outputs, not next-token probabilities. Actual models learn scores and values, use many heads and include residual paths and other transforms. This controlled fixture isolates what changing the mask alone can do.
Append Before Predicting Again
Read downward: the cat becomes the cat sat, then the cat sat down. Each bracket contains the prefix available for that step. The newly selected token joins the next prefix. Generation proceeds sequentially until a stop condition; causal training can calculate many shifted prediction positions in parallel because their gold prefixes are already supplied. The diagram does not assign a probability to down.
Where It Breaks
Return to the controlled attention row and change only the third value from 3 to 0, holding all scores fixed. The full-attention output changes from 1 to 0; the dashed amber extent marks the lost unit. The causal output stays 0 because its third weight is zero. With a new future value of 6, the same reasoning predicts full output 2 and causal output 0.
This is why a bidirectional state computed with future text cannot simply be treated as a causal prefix state. Original BERT's masked-token objective does not itself specify GPT's left-to-right joint probability or generation loop. Iterative mask filling or a different objective can generate text, so the limit is objective and information compatibility, not an absolute claim that an encoder cannot emit tokens. Neither attention pattern guarantees factual predictions.
The Quick Version
- GPT predicts shifted next-token targets from causal prefixes.
- Candidate probabilities describe uncertainty, not correctness.
- Causal attention includes the current position and earlier positions.
- BERT reconstructs selected tokens with context on both sides.
- Allowed attention pairs do not specify learned weight magnitudes.
- Both architectures can support task-trained classification heads.
- Unmasked future targets create a causal-training shortcut.
- A mask change alone can change a weighted blend.
- Autoregressive generation grows its prefix one token at a time.
- Bidirectional future-dependent states cannot be reused as causal prefixes.