Transformer Architecture

Stack blocks of self-attention plus a feed-forward network, each wrapped in a residual connection and normalization, and repeat that same block many times to build the whole model.

One transformer block is attention plus a feed-forward network, each wrapped in a residual connection and a normalization layer, and the whole block repeats N times

Why Does This Exist?

Every piece assembled so far in this band is a part, not a machine. Self-attention mixes information across positions. Positional encoding supplies the order that attention cannot see on its own. Feed-forward networks transform each position once its context is gathered. Alone, none of them is a working model: attention without depth cannot build hierarchical representations, and a feed-forward layer without attention cannot relate one token to another at all.

The 2017 contribution was not any single piece, recurrent and convolutional sequence models already existed. It was the specific way of combining attention and a feed-forward network into one repeatable block, wrapping that block so it trains reliably at real depth, and showing that stacking many copies outperformed the recurrent architectures that had dominated sequence modeling, while training far faster because attention parallelises across the whole sequence.

Think of It Like This

An assembly line where every station does the same two jobs

Picture an assembly line where every station performs the identical two-step process: first, consult with every other item currently on the line to gather relevant context, then apply a private transformation to what was gathered. The line has many stations, all running the same two-step process, but each station's specific weights are learned independently. Station 3 might gather very different context than station 20, even though the procedure at every station is identical.

That repeatable two-step station, copied end to end down the line, is the entire transformer architecture. Complexity comes from depth and learned specialization, not from each station doing something structurally different from the last.

Where it stops: a factory line passes each item forward once, while a transformer also hands every station its unmodified input through the residual path, so nothing a station does can destroy what arrived.

How It Actually Works

The block, assembled

The diagram above shows one complete transformer block. Two sub-layers, each wrapped identically:

x' = x + \text{SelfAttention}(\text{Norm}(x)), \qquad x'' = x' + \text{FFN}(\text{Norm}(x'))

This is the pre-norm ordering. Each sub-layer output is added back onto its own input through a residual connection, the same identity-plus-correction structure that made very deep CNNs trainable, applied here to make very deep transformer stacks trainable. Without it, dozens of stacked blocks would face the same degradation those connections were invented to fix.

What repeats, and what does not

The block structure, attention then feed-forward, each wrapped in a residual connection and normalization, is identical at every layer. The weights inside each block are entirely separate: layer 1's attention projections, layer 1's feed-forward matrices, layer 2's, and so on, are all independently learned. Stacking $N$ copies of this one block is what gives a transformer its depth, and $N$ is a genuine architectural choice. More blocks mean more capacity, at a roughly linear cost in compute and parameters per block.

Where positional encoding fits

Positional encoding is added exactly once, before the first block, directly to the input embeddings. It is not re-added at every layer. Once position is folded into the token representations, it propagates through the stack like any other information, carried along by the residual connections.

Encoder-only, decoder-only, and encoder-decoder

The block above is the general unit. How many stacks a model builds from it, and whether those stacks use causal masking, defines the three major variants. An encoder-only model in BERT's lineage stacks blocks with no causal mask, since it represents a complete input rather than generating one token at a time. A decoder-only model in GPT's lineage stacks blocks with causal masking throughout, since every position predicts the next token without peeking ahead. An encoder-decoder model runs both stacks, with the decoder additionally using cross-attention to read the encoder's output, the arrangement of the original translation paper. Decoder-only became the dominant choice for large language models, largely because one unified next-token objective scales cleanly to enormous unlabelled text.

Worked example

Normalization is the quiet half of the block, so watch it on $[2, 4, 4, 6]$ . The mean $m$ is $16/4 = 4$ . The variance $v$ is $(4 + 0 + 0 + 4)/4 = 2$ , so the spread is $\sqrt{2} \approx 1.4142$ . Subtract and divide:

\text{norm} = [-1.4142,\ 0.0,\ 0.0,\ 1.4142]

Add the stub correction at one tenth strength and the block returns $[1.8586,\ 4.0,\ 4.0,\ 6.1414]$ . Shape in, same shape out, which is exactly what lets $N$ blocks stack with no reshaping between them. The snippet below runs this.

Code

One transformer block in plain Python, with the residual additions kept visible and the same four numbers as the worked example.

1import math2
3
4def layer_norm(x: list[float], eps: float = 1e-5) -> list[float]:5    mean = sum(x) / len(x)6    var = sum((v - mean) ** 2 for v in x) / len(x)7    return [(v - mean) / math.sqrt(var + eps) for v in x]8
9
10def block(x: list[float]) -> list[float]:11    n = layer_norm(x)  # centre and scale first (pre-norm)12    return [v + 0.1 * w for v, w in zip(x, n)]  # stub correction plus the residual13
14
15x = [2.0, 4.0, 4.0, 6.0]16print([round(v, 4) for v in layer_norm(x)])  # -> [-1.4142, 0.0, 0.0, 1.4142]17print([round(v, 4) for v in block(x)])  # -> [1.8586, 4.0, 4.0, 6.1414]

A real block calls self-attention and the feed-forward network where the stub stands. The wiring, normalise then add back the input, is what the snippet is teaching.

Watch Out For

Assuming more blocks always means a better model

Stacking more transformer blocks increases capacity, but it also increases training difficulty, compute cost, and inference latency roughly proportionally. Past a certain depth for a given amount of training data, additional blocks stop paying for themselves: the model holds more capacity than the data can usefully fill, and each marginal block adds cost without adding quality. Depth is a real design choice with diminishing returns, not a dial to maximise by default.

Dropping the residual add when implementing from scratch

A common bug when hand-assembling a block is applying normalization and the sub-layer but forgetting to add the original input back, writing x = ffn(layer_norm(x)) instead of x = x + ffn(layer_norm(x)). It still runs and still produces the right shape, so nothing crashes, but it silently removes the connection that makes deep stacks trainable, and the model trains measurably worse with no error pointing at why. If a from-scratch stack will not train past a few layers, check the addition before touching anything else.

The Quick Version

A transformer block is self-attention plus a feed-forward network, each wrapped in a residual connection and normalization.
The block structure repeats identically across layers while the learned weights inside each layer stay separate.
Positional encoding is added once, before the first block, and rides the residuals from there.
Encoder-only, decoder-only, and encoder-decoder variants differ in which stacks exist and whether causal masking applies.
Decoder-only won large language models because next-token prediction scales cleanly to huge unlabelled corpora.
Normalising $[2, 4, 4, 6]$ gives $[-1.4142, 0, 0, 1.4142]$ , and every block preserves its input shape exactly.