Self-Attention

Why Does This Exist?

The attention mechanism that fixed sequence-to-sequence translation had a decoder looking back at an encoder — two different sequences, one reading the other. But nothing about "score, weigh, blend" actually requires two separate sequences. Nothing stops a sequence from attending to itself: every token scoring itself against every other token in the same sequence, including its own position.

That single reframing is self-attention, and it turns out to matter enormously. A recurrent network builds context about a sentence by passing a hidden state one token at a time — token 50's understanding of token 2 has to survive 48 sequential updates to arrive. Self-attention lets every token look directly at every other token in one step, regardless of how far apart they sit in the sequence. Distance stops being a cost.

Think of It Like This

How It Actually Works

Three vectors per token

Every input token's embedding $x_i$ is projected through three separate learned weight matrices into three roles:

The query ($q_i$) represents what token $i$ is looking for. The key ($k_i$) represents what token $i$ offers to be found by. The value ($v_i$) is the actual content token $i$ contributes if it's found relevant. All three come from the same token, but $W_Q$, $W_K$, $W_V$ are different matrices, so the three vectors end up capturing different things about it.

Score, scale, softmax, blend

For every query $q_i$, score it against every key in the sequence with a dot product, then scale, softmax, and blend against the values — the diagram above shows exactly this for token 2:

$QK^\top$ computes every query-key dot product at once — an $n \times n$ matrix of raw scores, one row per query, one column per key. Dividing by $\sqrt{d_k}$ (the key dimension) before the softmax keeps those scores from growing too large as $d_k$ increases; unscaled, large dot products push softmax toward a near one-hot spike and gradients toward zero. Softmax turns each row into weights that sum to 1, and multiplying by $V$ blends the value vectors by those weights — exactly the score-weigh-blend sequence from the earlier attention mechanism, just computed as one matrix expression covering every position in the sequence at once.

Why it's called "self"

In cross-attention, queries come from one sequence and keys/values come from a different one. In self-attention, $Q$, $K$, and $V$ all come from the same sequence — every token attends over the sequence it's a member of, itself included. That's the entire distinction, and it's why a single self-attention layer can build a representation of a token that already accounts for its entire surrounding context, in one operation.

What the weights actually encode

Attention weights are learned, not designed — nothing hand-specifies that pronouns should attend to their antecedents. The training signal (predicting the next token, or whatever the objective is) simply rewards weight patterns that help the prediction, and patterns like "attend to the subject of the sentence" turn out to help a great deal, so gradient descent finds them. The $QK^\top$ score is genuinely just a dot product between two learned vectors; the linguistic structure it ends up capturing is emergent, not engineered.

Show Me the Code

Scaled dot-product attention over a toy sequence of four tokens, with $Q$, $K$, $V$ built from one shared input and three separate projections.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max(axis=-1, keepdims=True))6    return e / e.sum(axis=-1, keepdims=True)7
8
9def self_attention(x: np.ndarray, w_q: np.ndarray, w_k: np.ndarray, w_v: np.ndarray) -> np.ndarray:10    q, k, v = x @ w_q, x @ w_k, x @ w_v          # (n, d_k) each, one row per token11    scores = q @ k.T / np.sqrt(k.shape[-1])       # (n, n) — every query against every key12    weights = softmax(scores)                     # each row sums to exactly 113    return weights @ v                             # (n, d_k) — blended output per token14
15
16rng = np.random.default_rng(0)17x = rng.normal(size=(4, 8))                        # 4 tokens, embedding dim 818w_q, w_k, w_v = (rng.normal(scale=0.3, size=(8, 4)) for _ in range(3))19output = self_attention(x, w_q, w_k, w_v)20print(output.shape)  # -> (4, 4) — one output vector per token, in the value dimension

Every row of scores before the softmax spans all four tokens — token 0's row includes its score against itself, at position 0, exactly as the diagram's diagonal shows.

Watch Out For

The Quick Version

• Self-attention projects every token into a query, a key, and a value using three separate learned matrices.
• Every query scores against every key via a dot product, scaled by $\sqrt{d_k}$, then softmaxed into weights summing to 1.
• Those weights blend the value vectors into one output per token, in a single matrix operation covering the whole sequence.
• "Self" means $Q$, $K$, and $V$ all come from the same sequence — contrast with cross-attention, where they come from two different sequences.
• Self-attention has no inherent sense of token order; that has to be injected separately.

What to Read Next

• Multi-Head Attention runs several self-attention computations in parallel, each in its own subspace.
• Causal Masking restricts self-attention so a position can't look ahead — what turns this mechanism into a generator.
• Attention Complexity measures the cost of the $QK^\top$ matrix this page builds.
• Attention Mechanism is the score-weigh-blend sequence self-attention specializes to one sequence attending to itself.
• Positional Encoding is the fix for the order-blindness this page's second pitfall names.