Transformer Architecture

Why Does This Exist?

Before transformers, models read sequences one step at a time, like RNNs and LSTMs. That caused two problems. Processing was sequential, so it was slow and hard to parallelize on modern hardware. And information from early words faded by the time the model reached the end of a long passage.

The transformer threw out step-by-step reading entirely. It looks at every token in the sequence at once and lets each one attend directly to every other, no matter how far apart they sit. That made training massively parallel and gave models a real grip on long-range context. It's the architecture behind essentially every large language model today.

Think of It Like This

A round-table meeting, not a game of telephone

Old sequence models were like a game of telephone: a message whispered down a line, degrading at every hop. A transformer is a round-table meeting where everyone hears everyone else directly and, in one pass, decides whose input matters most for the point at hand. Nothing has to survive a long relay.

How It Actually Works

• Tokenize and embed. Text is split into tokens, and each token becomes a vector called an embedding.
• Add positional encoding. Because the model sees all tokens simultaneously, position information is added so word order isn't lost.
• Run through stacked blocks. The same block is repeated many times, and each one contains:
  • Self-attention, where every token gathers context from the others. This is the engine, covered in its own page.
  • A feed-forward network, a small MLP applied at each position to process what attention gathered.
  • Residual connections and normalization, which keep gradients flowing and training stable.
• Choose a shape for the job. Translation-style tasks use an encoder plus a decoder; most LLMs are decoder-only, trained to predict the next token.

Inside one transformer block

Step 1 of 4

1Self-attention

Each token gathers context by attending to every other token in the sequence.

Prev

Next

Show all steps

1. Self-attention: Each token gathers context by attending to every other token in the sequence.
2. Add & normalize: A residual connection adds the input back in, then layer normalization keeps values stable.
3. Feed-forward: A small position-wise MLP transforms each token independently.
4. Add & normalize, then repeat: Another residual and norm — and the whole block stacks N times.

When to Use This

• You work with sequences such as text, code, audio, or image patches
• Long-range relationships in the input matter
• You can batch lots of data for parallel training
• You want to build on pretrained models like BERT or GPT-style LLMs

Watch Out For

Attention cost grows with the square of length

Every token attending to every other means compute scales with sequence length squared. Long documents get expensive fast, which is why context windows are limited and why efficient-attention variants exist.

Without positions, it's just a bag of words

Self-attention on its own has no sense of order. Strip out the positional encoding and "dog bites man" looks identical to "man bites dog."

The Quick Version

• Transformers process a whole sequence at once instead of step by step.
• Self-attention lets every token pull context directly from every other token.
• Each block pairs attention with a small feed-forward network, plus residuals and normalization.
• Positional encoding restores the word order that parallel processing would otherwise lose.
• It's the backbone of modern LLMs, at the cost of attention scaling quadratically with length.

What to Read Next

• Self-Attention is the mechanism that powers every transformer block.
• Neural Networks (Multi-Layer Perceptron) is the feed-forward piece inside each block.
• Definitions worth a look: Embedding, Attention, and Softmax.