Full Transformer Block

The Transformer block is the fundamental building block of modern large language models. It combines several critical layers into a single cohesive pipeline.

First, tokens are converted into embeddings, which are continuous vector representations of discrete words or subwords. Since transformers process all tokens in parallel, they have no inherent concept of order. To fix this, a positional encoding is added to each embedding, injecting a unique signal that depends on the token's position in the sequence.

The core of the block is multi-head attention. This layer allows each token to attend to other tokens in the sequence, gathering context and relationships. The attention mechanism computes a weighted sum of values based on the similarity between queries and keys. Splitting this into multiple heads allows the model to attend to different types of relationships simultaneously.

After attention, a residual connection adds the original input back to the attention output. This helps preserve the original information and allows gradients to flow easily during training, preventing the vanishing gradient problem. The result is then normalized using Layer Normalization, which stabilizes the activations and speeds up convergence.

Next, a feed-forward network (FFN) independently processes the representation of each token. While attention mixes information across the sequence, the FFN transforms the information within each token's representation.

Finally, another residual connection and layer normalization step is applied to the FFN output, completing the Transformer block. This pipeline of Attention -> Add & Norm -> FFN -> Add & Norm can be stacked many times to build deep, powerful models like GPT and BERT.