Visual explainer

Attention Mechanism

Follow a query from key scores through normalized weights to a value blend, then inspect how a new query and score scaling change it.

A fixed context reuses one summary for every output step.

Attention builds a query-specific blend of stored values. Earlier encoder-decoder models reused one fixed context; attention instead keeps source positions available for each query. Compressing a whole sentence into one summary can make long inputs harder, but no vector literally overflows.

Score Every Word

A query scores each key before values are mixed.

A query asks what to match; keys determine matching scores; values supply the material to blend. These illustrative vectors aren't learned word meanings. With four key coordinates, query [2,0,0,0] gives dots [4,2,0]; dividing by √4 gives scores [2,1,0].

Blend by Weight

Softmax weights sum to one and multiply the values.

Exponentiate the scores to [7.389,2.718,1], then divide by their sum, about 11.107. Weights round to [0.665,0.245,0.090]. Multiplying values [1,0], [0,1], [0,0] and adding gives output [0.665,0.245]. That output is a vector, distinct from the three weights.

Refocus Each Step

Changing the query changes the blend of the same values.

Keep the keys and values. Query [0,2,0,0] gives scores [0,1,2] and weights [0.090,0.245,0.665]. The output becomes [0.090,0.245], because the third value is [0,0]. Recompute weights for a new query; stored keys and values can still be reused.

Keep Source Positions Available

Attention keeps source positions available to each query.

The decoder can address individual encoder positions rather than depend entirely on one shared summary. This cross-attention example uses decoder queries and encoder keys/values. In self-attention, all three originate from the same sequence. Access doesn't guarantee useful matching or perfect recall; scoring more positions costs more work.

Shrink the Scores

Divide dot products by the square root of key width.

Our four-coordinate keys use divisor 2. With 64 coordinates, the divisor would be 8. Under independent, zero-mean, unit-variance query/key components, scaling counters dot-product variance growth. It doesn't guarantee broad weights; large score differences can remain.

Where It Breaks

Larger score gaps can make attention sharply concentrated.

Without scaling, our dots [4,2,0] produce weights about [0.867,0.117,0.016]. Every weight remains positive. Much larger gaps can make local softmax derivatives tiny; finite scores don't imply exact one-hot weights or zero gradients throughout the model. Sharp attention also needn't identify the right source.

The Quick Version

Fixed-context decoders reuse one summary across output steps.
Queries score keys; values supply the blend's contents.
Softmax weights sum to one before multiplying value vectors.
New queries change weights while keys and values can remain.
Access to source positions doesn't guarantee perfect recall.
Scale dot products by the square root of key width.
Large score gaps can concentrate weights and reduce softmax sensitivity.