Visual explainer
Transformers
Trace positioned token rows through attention heads, residual normalization and a feed-forward network, then inspect decoder masks and dense pair growth.
A Transformer builds contextual token representations by combining attention with a separate channel transformation at each position. Attention lets known input rows be processed without the position-to-position recurrent state dependency drawn above. Layers still depend on earlier layers, and autoregressive generation still depends on previously generated tokens.
The lower bus represents the shared bank of known input rows, not a shared output or one broadcast value. Each destination computes a different weighted mixture later. This is a dependency comparison, not a latency benchmark or a claim that every recurrent architecture compresses input into one vector. The original Transformer uses an encoder and decoder; the small fixture below illustrates its post-normalization block arrangement with dropout disabled.
Supply an Order Signal
An embedding is a row of numbers representing a token. The picture uses repeated token A with embedding [1,0,0,0] at positions zero and two. Add illustrative positional vectors [0,0,0,0] and [0,0,0,2]: the fourth channel changes while the token's first channel stays fixed. Bars rise 50 drawing units per feature unit; dashes mark zero. This controlled repeated-token diagnostic makes the order signal visible.
For the main three-token fixture, use embeddings A = [1,0,0,0], B = [0,1,0,0], C = [0,0,1,0] and position vector [0,0,0,p] at position p = 0,1,2. The resulting input matrix X has rows [1,0,0,0], [0,1,0,1], [0,0,1,2]. These chosen embedding values absorb any embedding scale. The position vectors illustrate a learned-position choice, not the original paper's sine and cosine formulas. That paper discusses both fixed and learned encodings.
Blend One Attention Row
Each head makes a query describing what to match, keys supplying matching features, and values supplying contents. For the first token in head one, use query [2√2,0], keys A = [1,0], B = [0.5,0.5], C = [0,1], and values A = [1,0], B = [0,1], C = [0,0]. The next picture shows the learned projections that construct them.
Dots [2√2,√2,0] divided by √2, the square root of head width two, give scores [2,1,0]. Exponentials [7.389056,2.718282,1] sum to 11.107338; division gives weights [0.665240956,0.244728471,0.090030573]. The strip is one unit partitioned in those shares. Rounded displayed numbers approximate the exact normalization.
Multiplying values and adding gives [0.665240956,0.244728471], the diamond inside the value triangle. Horizontal position is channel one and vertical height is channel two; C is zero, A is [1,0], and B is [0,1]. The diamond is a two-channel vector, not a three-element weight row. Compatibility does not establish factual correctness or a guaranteed linguistic interpretation. The attention mechanism visual develops scoring, masks and this blend in detail.
Project Matching and Contents
The left chart shows first-token row [1,0,0,0]; the right charts show two channels each for Q, K and V. All use a vertical scale of 24 drawing units per feature unit. Zero components remain dashes. The branch arrows represent different learned linear maps, not probability magnitudes.
Our model width is four; each of two heads has key and value width two. Every per-head projection is therefore a four-by-two matrix. With row-vector multiplication, choose head one's query matrix rows as [[2√2,0],[0,2√2],[√2,√2],[0,√2]]. Choose key matrix rows [[1,0],[0.5,0.5],[0,1],[0,0]] and value matrix rows [[1,0],[0,1],[0,0],[0,0]] for both heads. Multiplying the first input row selects the first matrix row: Q = [2√2,0], K = [1,0], V = [1,0].
The complete projected key rows are [1,0], [0.5,0.5], [0,1]; value rows are [1,0], [0,1], [0,0]. Head one's query rows are [2√2,0], [0,3√2], [√2,3√2], yielding scaled score rows [2,1,0], [0,1.5,3], [1,2,3]. The same positioned X supplies every projection. These synthetic matrices make the arithmetic checkable; learned axes are not fixed human concepts.
Join the Heads
Head two uses query-matrix rows [[0,2√2],[2√2,0],[√2,√2],[√2,0]], retaining the same key and value matrices. Its first-token query [0,2√2] reverses scores to [0,1,2], producing output [0.090030573,0.244728471]. Head one's output is [0.665240956,0.244728471].
The top charts use 80 drawing units per feature unit. Below, their two-channel outputs join in head order as [0.665240956,0.244728471,0.090030573,0.244728471]. Concatenation preserves four separate channels; it does not average the heads. An output projection then maps the joined channels back to model width. Choose identity four-by-four output projection WO here so these numbers remain unchanged.
Other token rows also have independent head outputs: head one gives [0.039112573,0.175290392] and [0.090030573,0.244728471]; head two gives [0.785597035,0.175290392] and [0.665240956,0.244728471]. Real head and output projections are learned. A head has no guaranteed grammar job, and two heads do not guarantee better results for every task.
Add Before Normalizing
The bypass carries original input X around the complete multi-head operation. The box labelled heads includes head concatenation and WO projection before the addition. For the first token, add [1,0,0,0] to the four projected channels to obtain residual row [1.665240956,0.244728471,0.090030573,0.244728471].
Layer normalization takes statistics across one token's channels. Their mean is 0.561182118; their population variance, the mean squared deviation over four channels, is 0.410303879. With epsilon 0.00001, gain gamma = one and bias beta = zero, normalize each channel as (channel − mean) / √(variance + epsilon). The resulting row is [1.723590592,−0.494028497,−0.735533598,−0.494028497].
The signed bars show those channels at 45 drawing units per feature unit: upward is positive, downward negative, and the horizontal line is zero. This is post-normalization, the original arrangement; other Transformer variants place normalization differently. Learned gain and bias can change the resulting mean and variance, and epsilon means even this unit-gain fixture's variance is slightly below one. Normalization here does not combine different token rows or use batch statistics.
Transform Each Token's Channels
Attention mixes information across token positions; a position-wise feed-forward network mixes channels separately within each token. The picture follows one selected row from the previous scene. The same FFN parameters act independently on every row; there is no B-to-A or C-to-A mixing in this operation. Bars use a 40-unit vertical scale, with dashes for zero.
For a small checkable network, let W1 concatenate the four-by-four identity matrix with its negative, so an input row x expands to [x,−x]. Set both biases to zero. ReLU replaces negative activations with zero. The eight middle activations are [1.723590592,0,0,0,0,0.494028497,0.735533598,0.494028497]. Let W2 take only their first four channels, using identity above a zero matrix. The output becomes [1.723590592,0,0,0]. This sparse choice demonstrates expansion, nonlinearity and projection; it claims no quality improvement.
The original block also adds the FFN's input as a second residual, then normalizes again. The summed first-token row is [3.447181184,−0.494028497,−0.735533598,−0.494028497]. Using the same normalization settings gives mean 0.430897648, variance 3.042376242, and final row [1.729278689,−0.530273448,−0.668731792,−0.530273448]. Thus the bare FFN output in the image is not the complete block output.
Bridge Encoder and Decoder
This schematic switches from the numeric encoder-block detail to the original encoder-decoder architecture. The encoder uses source self-attention followed by a position-wise FFN; each sublayer has its residual and normalization. Its output preserves one row per source position, drawn as separate lines. Repeated layers are grouped into the encoder box.
The decoder first applies masked self-attention to shifted target rows. Its resulting rows supply cross-attention queries Q; encoder outputs supply keys and values KV. Cross-attention can therefore retrieve source information while the decoder builds target representations. A decoder FFN follows. Residuals and normalizations also surround these decoder sublayers; they are omitted from this overview's paths, not removed from the architecture.
The connections indicate origins, not measured tensor magnitudes or new toy outputs. Decoder-only models have a different arrangement and do not acquire an encoder simply because they are Transformers. The BERT versus GPT visual compares encoder-style and decoder-style visibility.
Predict Without Future Targets
For a target sequence A, B, C, train on shifted decoder inputs BOS, A, B, where BOS marks the beginning of sequence. Each grid row is one input query, each column an input key. Dots are allowed positions; crosses are excluded future positions. The first row predicts A from BOS, the second predicts B from BOS and A, and the third predicts C from BOS, A and B. The letters at right are training targets, not invented model predictions or probabilities.
The highlighted diagonal is valid: the second row reads its input A while predicting B, not A. Add negative infinity to scores above the diagonal before row softmax, leaving weights zero on forbidden keys. The three rows have one, two and three allowed positions. Known shifted training rows can be processed together, while generation must produce a token before appending it to the input for the next step.
After decoder layers, a learned linear vocabulary projection and softmax produce next-token probabilities. That vocabulary operation is separate from the attention softmax over source positions. This schematic supplies no numeric vocabulary logits or guarantee that the target will be predicted correctly.
Where It Breaks
Each cell is one query-key pair at a common cell size. Three tokens require 3×3 = 9 dense self-attention pairs; six require 6×6 = 36. The side doubles and area quadruples. With fixed head width, dense pair arithmetic therefore grows quadratically with sequence length. The amber square pictures that cost boundary, not a guaranteed out-of-memory event.
Storing a naive full score matrix also requires quadratic entries per head, but not every Transformer allocation or implementation has that memory behavior. For example, retaining one score matrix for every combination of 4,096 tokens, 32 heads and 32 layers at two bytes per entry would require 4096² × 32 × 32 × 2 = 34,359,738,368 bytes, or 32 GiB. This is a hypothetical retained-score allocation, not a measured inference peak.
FlashAttention computes exact attention with tiling to avoid materializing the full score matrix in high-bandwidth memory; it does not eliminate dense pair arithmetic. A restricted local window changes which pairs are allowed, trading direct long-range access for fewer interactions. Prefer restricted visibility only when that lost access fits the task; exact tiling and sparse visibility solve different constraints.
The Quick Version
- Self-attention removes recurrence between known input positions.
- Positional information lets token rows carry an order signal.
- Each attention row normalizes weights before blending values.
- Learned projections construct separate query, key and value channels.
- Head outputs concatenate before a learned output projection.
- The original block adds residuals before channel normalization.
- A shared feed-forward mapping transforms each token's channels separately.
- Cross-attention bridges decoder queries to encoder keys and values.
- Shifted inputs and causal masks prevent future-target access.
- Doubling sequence length quadruples dense attention pairs.