Transformer Architecture Cheat Sheet

The stack, top to bottom

A transformer block is a repeatable unit. Stack N of them and you have the encoder (or decoder) trunk.

1. Token + positional embeddings turn token ids into vectors and inject order.
2. Multi-head self-attention lets every token look at every other token.
3. Add & Norm wraps each sublayer in a residual connection followed by LayerNorm.
4. Feed-forward network applies a position-wise MLP (usually 4x the model width).
5. Add & Norm again, then hand off to the next block.

Attention in one line

For queries Q, keys K, and values V:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

The √dₖ scaling keeps the dot products from growing too large and saturating the softmax.

Shapes to remember

Tensor	Shape
Input ids	(batch, seq)
Embeddings	(batch, seq, d_model)
Attention scores	(batch, heads, seq, seq)
Block output	(batch, seq, d_model)

Gotchas

• Decoder self-attention is masked so a position can't attend to the future.
• Multiple heads split d_model into heads × d_head; they don't add parameters over one wide head.
• LayerNorm placement (pre- vs post-norm) meaningfully changes training stability.