Transformer Architecture

Why Does This Exist?

Every piece assembled so far in this band is a part, not a machine. Self-attention mixes information across positions. Positional encoding supplies the order self-attention can't see on its own. Feed-forward networks transform each position's content once it's gathered. None of these, alone, is a working model — self-attention without depth can't build hierarchical representations, and a single feed-forward layer without attention has no way to relate one token to another at all.

The 2017 contribution wasn't any one of these pieces in isolation — recurrent and convolutional sequence models already existed. It was the specific way of combining attention and a feed-forward network into one repeatable block, wrapping that block so it trains reliably at real depth, and showing that stacking many copies of it outperformed the recurrent architectures that had dominated sequence modeling until then, while training dramatically faster because attention parallelizes across the whole sequence.

Think of It Like This

How It Actually Works

The block, assembled

The diagram above shows one complete transformer block. Two sub-layers, each wrapped identically:

(This is the pre-norm ordering — pre-norm vs post-norm covers the alternative and why it matters.) Each sub-layer's output is added back onto its own input via a residual connection — the same identity-plus-correction structure that made very deep CNNs trainable, applied here to make very deep transformer stacks trainable. Without it, a stack of dozens of blocks would face the same degradation problem residual connections were invented to fix in the first place.

What repeats, and what doesn't

The block's structure — attention, then feed-forward, each wrapped in a residual connection and normalization — is identical at every layer. The weights inside each block are entirely separate: layer 1's attention projections, layer 1's feed-forward matrices, layer 2's attention projections, and so on, are all independently learned parameters. Stacking $N$ copies of this one block is what gives a transformer its depth, and $N$ is a genuine architectural choice — more blocks generally mean more capacity, at a roughly linear cost in compute and parameters per additional block.

Where positional encoding fits

Positional encoding is added exactly once, before the first block, directly to the input embeddings. It is not re-added at every layer — once position is folded into the token representations, it propagates through the stack the same way any other information does, carried along by the residual connections.

Encoder-only, decoder-only, and encoder-decoder

The block described above is the general unit; how many stacks a model builds from it, and whether those stacks use causal masking, defines the three major variants. An encoder-only model (BERT's lineage) stacks blocks with no causal mask, since it's building a representation of a complete input rather than generating one token at a time. A decoder-only model (GPT's lineage) stacks blocks with causal masking throughout, since every position is trained to predict the next token without peeking ahead. An encoder-decoder model runs both stacks, with the decoder additionally using cross-attention to read the encoder's output — the architecture the original transformer paper introduced for translation. Decoder-only became the dominant choice for large language models, largely because a single unified training objective (next-token prediction, everywhere) scales cleanly to enormous unlabeled text corpora in a way that made it the practical winner.

Show Me the Code

One transformer block, composing the pieces from earlier pages, with the residual connections kept visible.

1import numpy as np2
3
4def layer_norm(x: np.ndarray, eps: float = 1e-5) -> np.ndarray:5    mean, var = x.mean(axis=-1, keepdims=True), x.var(axis=-1, keepdims=True)6    return (x - mean) / np.sqrt(var + eps)7
8
9def transformer_block(x: np.ndarray, attn_fn, ffn_fn) -> np.ndarray:10    x = x + attn_fn(layer_norm(x))     # residual + pre-norm around attention11    x = x + ffn_fn(layer_norm(x))       # residual + pre-norm around the feed-forward net12    return x13
14
15def stub_attention(x: np.ndarray) -> np.ndarray:16    return x * 0.1                       # stand-in: a real block calls self-attention here17
18
19def stub_ffn(x: np.ndarray) -> np.ndarray:20    return np.tanh(x) * 0.1              # stand-in: a real block calls the FFN here21
22
23x = np.random.default_rng(0).normal(size=(4, 8))24for layer in range(3):                    # stack N=3 identical-structure blocks25    x = transformer_block(x, stub_attention, stub_ffn)26print(x.shape)  # -> (4, 8) — shape is preserved through every block, by construction

Every block preserves the input shape exactly — that's what lets $N$ identical-structure blocks stack without any reshaping logic between them, regardless of how many layers deep the model goes.

Watch Out For

The Quick Version

• A transformer block is self-attention plus a feed-forward network, each wrapped in a residual connection and normalization.
• The block's structure repeats identically across layers; the learned weights inside each layer are entirely separate.
• Positional encoding is added once, before the first block, and propagates through the stack via the residual connections.
• Encoder-only, decoder-only, and encoder-decoder variants differ in which stacks exist and whether causal masking is applied.
• Decoder-only became the dominant choice for large language models because next-token prediction scales cleanly to large unlabeled corpora.

What to Read Next

• Self-Attention is the first of the two sub-layers this page's block assembles.
• Feed-Forward Networks is the second sub-layer, and the one that does per-position transformation.
• Pre-Norm vs Post-Norm covers the exact placement of normalization this page's formula uses.
• Positional Encoding supplies the order signal this architecture adds once, before the first block.
• Multi-Head Attention is the actual attention sub-layer used inside each block in practice.