Encoder-Decoder Architectures
The three ways to arrange transformer blocks — encoder-only for bidirectional understanding, decoder-only with causal masking for generation, and encoder-decoder stacks for cross-attention sequence mapping.
Think of It Like This
Like having an interpreter read a whole foreign speech into clear shorthand notes before passing them to a writer who crafts the final translation.
Same transformer block, three skeletons. Encoder-only stacks have no causal mask and build a representation of a complete input — BERT's lineage. Decoder-only stacks mask every position causally and generate one token at a time — GPT's lineage. Encoder-decoder runs both, with the decoder cross-attending into the encoder's output, the original design built for translation.
Decoder-only became the dominant large-model choice because next-token prediction scales to unlabeled text in a way paired input-output data never could.