Skip to content
AI360XpertBeta
Glossary
Definition

Encoder-Decoder Architectures

The three ways to arrange transformer blocks — encoder-only for bidirectional understanding, decoder-only with causal masking for generation, and encoder-decoder stacks for cross-attention sequence mapping.

Think of It Like This

Like having an interpreter read a whole foreign speech into clear shorthand notes before passing them to a writer who crafts the final translation.

Same transformer block, three skeletons. Encoder-only stacks have no causal mask and build a representation of a complete input — BERT's lineage. Decoder-only stacks mask every position causally and generate one token at a time — GPT's lineage. Encoder-decoder runs both, with the decoder cross-attending into the encoder's output, the original design built for translation.

Decoder-only became the dominant large-model choice because next-token prediction scales to unlabeled text in a way paired input-output data never could.