Causal Masking

Causal masking hides every future token before attention weights are computed, so each position can only blend in itself and the tokens before it.

A causal mask adds negative infinity to future scores so row two turns 1.0 and 3.0 into weights 0.119 and 0.881 with the future pinned at zero

Why Does This Exist?

Start with self-attention. Each token builds three vectors: a query that asks, a key that advertises, and a value that carries content. It scores its query against every key, runs softmax, and blends the values. Left alone, that lookup sees the whole sequence at once, past and future together.

That full view breaks next-token training. Say you train a model to predict token 4 from the representation at token 3, but token 3 was allowed to attend to token 4. The model does not learn grammar or facts. It learns to copy the answer it already saw. Training loss looks excellent and means nothing. At generation time token 4 does not exist yet when you need to predict it, so the trick collapses the moment you use the model.

In our three-token example the raw future score is 5.0, larger than the true past scores 1.0 and 3.0. Without a mask the future would take 0.867 of the weight. The mask is what stops that theft.

This page covers the mask itself: the added matrix, the triangle it leaves, and the hand-computed weights. GPT is the decoder-only model built from this mask. Next-token prediction is the training job the mask keeps honest. BERT is the opposite choice, an encoder with no mask that reads both directions. Cross-attention is the decoder's other lookup, reading a separate sequence, so it needs no causal mask.

Think of It Like This

The mystery novel with glued pages

You read a mystery where every future chapter is glued shut. You can flip back and re-read any chapter you finished, and you can stare at the page you are on, but you cannot peel open what comes next. To guess the next page you only get the past plus the present.

That is the mask. Glued pages are the future positions set to negative infinity. Open pages are the allowed positions set to zero. Your guess blends only the open pages, the same way attention blends only unmasked values.

Where it stops: glue does not change the printed words, but attention rewrites each token from the tokens it was allowed to see. And a novel has one fixed future, while a model faces many possible next tokens at every step.

How It Actually Works

Attention takes three matrices. $Q$ holds queries, one row per position asking what to look for. $K$ holds keys, one row per position advertising what it offers. $V$ holds values, the content you blend. $d_k$ is the width of one query or key vector, used only to keep dot products from growing with width. $M$ is the causal mask, an $n$ by $n$ grid for a sequence of length $n$ . $i$ names the query row and $j$ names the key column.

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V

Here $M_{ij} = 0$ when $j \le i$ and $M_{ij} = -\infty$ when $j > i$ . Adding $0$ leaves a score alone. Adding $-\infty$ sends it to negative infinity, and $\exp(-\infty) = 0$ , so softmax gives that position exactly zero weight. Because the $-\infty$ lands before softmax, the surviving weights still sum to 1.

The mask rule in one grid

Keep position $(i, j)$ when the key $j$ sits at or before the query $i$ , block it when the key sits after the query. The kept cells form the lower triangle including the diagonal. The diagonal always stays open, so every token attends to itself. The whole upper triangle stays blocked, so no query ever sees a later key. The diagram draws this grid with the blocked cells in black and the worked row outlined.

Worked example: row two turns 1.0 and 3.0 into 0.119 and 0.881

Use three positions with scores already scaled, so you can follow the softmax by hand. The raw scores put 5.0 on the future, on purpose, to prove the mask wins over large values:

\text{scores} = \begin{bmatrix} 2.0 & 5.0 & 5.0 \\ 1.0 & 3.0 & 5.0 \\ 1.0 & 2.0 & 3.0 \end{bmatrix}

Add the mask. Future entries become $-\infty$ :

\text{masked} = \begin{bmatrix} 2.0 & -\infty & -\infty \\ 1.0 & 3.0 & -\infty \\ 1.0 & 2.0 & 3.0 \end{bmatrix}

Row one has one option. $\exp(2.0) \approx 7.389$ alone, so the weights are $[1.000, 0, 0]$ .

Row two is the worked row. Exponentiate the two survivors:

\exp(1.0) \approx 2.718, \quad \exp(3.0) \approx 20.086, \quad \text{sum} \approx 22.804

w_1 = \frac{2.718}{22.804} \approx 0.119, \quad w_2 = \frac{20.086}{22.804} \approx 0.881

So row two gives $[0.119, 0.881, 0]$ . The future weight is exactly 0, not small. Without the mask the same row would softmax $[1.0, 3.0, 5.0]$ into $[0.016, 0.117, 0.867]$ , and the future would steal 0.867. That gap between 0.867 and 0 is the whole point.

Row three is fully visible. $\exp(1.0) \approx 2.718$ , $\exp(2.0) \approx 7.389$ , $\exp(3.0) \approx 20.086$ , sum $\approx 30.193$ , so the weights are $[0.090, 0.245, 0.665]$ . The diagram shows these same six numbers: scores 1.0 and 3.0 on the left, weights 0.119 and 0.881 on the right.

Why the triangle trains every position at once

The mask applies per row instead of cutting the input short. Position 2 predicts from positions 1 to 2, position 3 predicts from positions 1 to 3, and both predictions compute in the same forward pass. Every position supplies one fair next-token label, none of them peeked, so a 1,000-token page gives 1,000 training signals at once. That parallelism plus the no-peeking guarantee is why decoder-only models scaled so well. At generation time the same rule holds: each new token only ever sees what came before it.

Code

The point is the check: build the mask, add it before softmax, and confirm the future lands at exactly zero even though its raw 5.0 is the largest score.

1import numpy as np2
3def softmax(z: np.ndarray) -> np.ndarray:4    e: np.ndarray = np.exp(z - np.max(z, axis=-1, keepdims=True))5    return e / np.sum(e, axis=-1, keepdims=True)6
7def causal_mask(n: int) -> np.ndarray:8    upper: np.ndarray = np.triu(np.ones((n, n)), k=1)9    return np.where(upper == 1, -np.inf, 0.0)10
11scores: np.ndarray = np.array([12    [2.0, 5.0, 5.0],13    [1.0, 3.0, 5.0],14    [1.0, 2.0, 3.0],15])16weights: np.ndarray = softmax(scores + causal_mask(3))17print(np.round(weights, 3))18# -> [[1.    0.    0.   ]19#     [0.119 0.881 0.   ]20#     [0.09  0.245 0.665]]

Row one keeps all weight on itself. Row two splits 0.119 and 0.881 across the past only. The 5.0 future scores contribute nothing, even though they are the largest raw numbers in their rows.

Watch Out For

Zeroing weights after softmax instead of masking before

Softmax first, then setting future weights to zero, looks the same but breaks the math. Take row two raw $[1.0, 3.0, 5.0]$ . Correct masking gives $[0.119, 0.881, 0]$ summing to 1. Late zeroing softmaxes the full row into $[0.016, 0.117, 0.867]$ , then blanks the last entry to leave $[0.016, 0.117, 0]$ , which sums to 0.133.

Symptom: rows that sum well below 1, outputs that look muted, and a bug that survives casual inspection because the future entries do read zero. Fix: always add negative infinity to the scores before softmax, never touch the weights after.

Putting a causal mask on an encoder that needs full context

A causal mask is for generation, not for labeling a complete input. An encoder for sentiment, entailment, or span tagging should read left and right at once, the way BERT does.

Symptom: accuracy drops on jobs where the right context decides the label, like negation or coreference, and a bigger model does not fix it because the information was hidden by design. Fix: leave encoders unmasked, keep the causal mask in decoders, and check which stack you edited when you assemble a transformer from parts.

The Quick Version

Causal masking adds negative infinity to every score where the key sits after the query, and zero elsewhere.
After softmax those future positions get exactly zero weight, so row two turns scores 1.0 and 3.0 into weights 0.119 and 0.881.
The allowed pattern is a lower triangle including the diagonal, so each token sees itself and every earlier token.
Without the mask the 5.0 future score in row two would take 0.867 of the weight, letting training copy the answer.
Masking must happen before softmax, or the surviving weights will not sum to 1.
Causal masks belong in decoders that generate, while encoders that label complete inputs stay unmasked.