Self-Attention

Every token builds a query, a key, and a value, then scores its query against every key in the sequence to decide how much of each other token's value to blend in.

Scores of 2.0, 1.0 and 0.0 become weights 0.665, 0.245 and 0.090 that blend values 10, 20 and 30 into the single output 14.25

Why Does This Exist?

The attention that fixed translation had a decoder looking back at a separate encoder. One sequence read another. Yet nothing in score, weigh, blend demands two sequences. A sequence can attend to itself, with every token scoring itself against every other token including its own position.

That reframing matters because recurrence pays for distance. A recurrent net carries context about token 2 forward through dozens of sequential updates before token 50 can use it, and long trips degrade the signal. Self-attention lets every token read every other token in one step, near or far. Distance stops costing steps. The two sequence setup it grew from is in attention-mechanism, the multi head extension is in multi-head-attention, and the order signal it still needs comes from positional-encoding.

Think of It Like This

A room where everyone hears everyone

Picture a meeting where information moves person to person around a table. To learn what the first speaker said, the last speaker waits through every relay in between, and each retelling risks losing a detail.

Now let every person hear every other person directly and at once, with each listener turning up the volume on whichever voices matter for their current question. Someone resolving the word it listens hardest to the earlier phrase that fixes its meaning, without waiting for relays.

That is self-attention: every position hears every position and weights each by relevance to itself, right now.

Where it stops: people choose what to amplify with intent, while tokens amplify whatever the trained weights score highly. The mechanism also hears all positions at once with no sense of who spoke first, so order must be supplied separately.

How It Actually Works

Each token embedding $x_i$ is projected through three learned matrices into three roles:

q_i = x_i W_Q, \qquad k_i = x_i W_K, \qquad v_i = x_i W_V

Here $q_i$ is the query, what token $i$ looks for. The term $k_i$ is the key, what token $i$ offers to be found by. The term $v_i$ is the value, the content token $i$ contributes when found relevant. All three grow from the same token, but $W_Q$ , $W_K$ and $W_V$ differ, so each role captures something different about it.

Score, scale, softmax, blend

Every query scores against every key at once:

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V

The product $QK^T$ holds every query key dot product in one $n \times n$ table, one row per query and one column per key. Dividing by $\sqrt{d_k}$ , with $d_k$ the key width, keeps scores from ballooning as widths grow. Unscaled large scores drive softmax toward a single spike and starve gradients. Softmax turns each row into weights summing to $1$ , and multiplying by $V$ blends the value rows by those weights.

Why it is called self

In cross-attention, queries come from one sequence while keys and values come from another. Here $Q$ , $K$ and $V$ all come from the same sequence, every token attending over its own sequence including itself. One layer therefore builds each token a representation already mixed with its full surrounding context.

Worked example

Take one query facing three keys with raw scores $[2.0, 1.0, 0.0]$ . Exponentiate to $[7.389, 2.718, 1.0]$ , where $7.389$ is $e^2$ and $2.718$ is $e^1$ rounded to three decimals. The sum is $7.389 + 2.718 + 1.0 = 11.107$ . Divide to get weights $[0.665, 0.245, 0.090]$ , since $7.389 / 11.107 = 0.665$ , $2.718 / 11.107 = 0.245$ and $1.0 / 11.107 = 0.090$ . They sum to $1.0$ .

Now blend values $[10.0, 20.0, 30.0]$ with those weights: $0.665 \times 10.0 + 0.245 \times 20.0 + 0.090 \times 30.0 = 6.65 + 4.9 + 2.7 = 14.25$ . The top score takes about two thirds of the weight without taking all of it, which is why attention blends rather than selects. The diagram draws these exact bars and the $14.25$ they produce.

Code

Scaled dot product attention reduced to one row, using only the standard library. It recomputes the worked example exactly.

1import math2
3scores = [2.0, 1.0, 0.0]4exps = [math.exp(s) for s in scores]5total = sum(exps)6weights = [e / total for e in exps]7print([round(w, 3) for w in weights])  # -> [0.665, 0.245, 0.09]8
9values = [10.0, 20.0, 30.0]10print(round(sum(w * v for w, v in zip(weights, values)), 2))  # -> 14.25

The printed weights read $0.665$ , $0.245$ and $0.09$ , the last being $0.090$ with its trailing zero dropped by printing. The blend lands on $14.25$ , matching the hand computation.

Watch Out For

Dropping the root d scale and blaming the optimizer

The symptom looks like a learning rate problem. Loss barely moves from the start, and lowering the rate changes nothing. The real cause sits upstream: without division by $\sqrt{d_k}$ , wide keys produce large raw scores, softmax saturates near a single $1$ , and gradients through it collapse toward zero almost everywhere.

Check for the scale factor before touching hyperparameters. It belongs inside the score computation, applied before every softmax.

Assuming the mechanism knows the order

The symptom is a model treating shuffled word orders as equivalent. Self-attention is permutation equivariant on its own: reorder the inputs and the same outputs return in the new order, because no position enters the dot products.

That is not a bug inside attention. Order has to be injected separately before attention runs, which is the entire job of positional encodings. If swaps go unnoticed, the position signal is missing rather than the attention broken.

The Quick Version

Every token is projected into a query, a key and a value through three separate learned matrices.
Each query scores against every key with a dot product scaled by $\sqrt{d_k}$ , then softmaxed into weights that sum to $1$ .
Those weights blend the value vectors, so scores $[2.0, 1.0, 0.0]$ become $[0.665, 0.245, 0.090]$ and values $[10.0, 20.0, 30.0]$ become $14.25$ .
Self means $Q$ , $K$ and $V$ share one sequence, unlike cross-attention across two sequences.
The mechanism carries no order on its own, so position must be added separately before it runs.