Encoder-Decoder Architectures
Three ways to arrange the same transformer block: read everything at once, generate one token at a time, or bridge both with cross-attention.
Why Does This Exist?
The transformer defines one repeatable block: attention, then a feed forward network, each wrapped in a residual. But a block is not a model. You still decide how many stacks to build, whether a position may look at tokens ahead of it, and whether one stack may read another stack output. Those three choices build three genuinely different machines from identical parts.
The 2017 paper answered all three one way for translation: an encoder stack that reads the whole source sentence with no restriction, plus a decoder stack that writes the target one word at a time while reading the encoder. That is the encoder-decoder layout. Drop the decoder and you get an encoder-only model that understands a complete input instead of generating, which is BERT lineage. Drop the encoder and keep only the masked decoder and you get a decoder-only model that generates token after token, which is GPT lineage. Same block, three skeletons, three jobs.
This page maps the three and shows what the mask actually blocks. The block internals live in transformers, the reading mechanism lives in cross-attention and causal masking, and the task family that needs both stacks lives in sequence-to-sequence.
Think of It Like This
One parts room, three assembly lines
A factory stocks identical workstations. The first manager lays them in one line where every station sees the whole order sheet before starting. Great for grading and sorting finished orders, useless for building one from scratch.
The second manager lays the same stations in a line where each station sees only what earlier stations finished, never the finished product. Perfect for building step by step. The third manager builds both lines and adds a conveyor so the building line can check the finished order sheet at every step, which is the right setup when the job is turning one specific input into a different output.
Same stations everywhere. What differs is the wiring: how many lines run, whether a station may see ahead, and whether one line reads the other work.
Where it stops: factory lines are fixed hardware, while these layouts are masking choices over learned weights, and the conveyor here is learned attention with queries, not a fixed belt. The shapes match, the learning does not.
How It Actually Works
Encoder-only: no restriction, built to understand
Encoder-only stacks blocks with no causal mask. Every position attends to every other position, before and after it. That fits jobs that need one good representation of a complete input: sentence sentiment, named entities, sentence pair scoring. Nothing is generated, so nothing needs hiding. BERT is the canonical member, and it stays the right tool whenever the output is a judgment rather than text.
Decoder-only: masked throughout, built to generate
Decoder-only masks every position causally: it sees itself and everything before, never anything after. That restriction is what makes next token training coherent. Predict the next token at every position, and at inference time extend one step at a time, never needing the ending in advance because training never showed it either. GPT is the canonical member.
Encoder-decoder: both stacks, bridged by cross-attention
Encoder-decoder runs an unmasked encoder over the source and a masked decoder that generates, with the decoder running cross-attention into the encoder output at every layer. Queries come from the decoder, keys and values from the encoder. That is the 2017 design, still the natural fit when the output transforms a specific input: translation, summarization, text to text models.
Worked example: what a 4 token mask really blocks
Take 4 tokens. The attention grid is 4 by 4, so 16 cells, where row is the reading position and column is the read-from position. Encoder-only blocks nothing: 0 of 16. Decoder-only blocks every cell with ahead of , meaning strictly above the diagonal: row 1 blocks 3, row 2 blocks 2, row 3 blocks 1, row 4 blocks 0. Add them:
So 6 of 16 cells are blocked and 16 minus 6 is 10 visible. Position 3 sees tokens 1, 2, and 3, exactly the prefix it needs and nothing else. That is the diagram middle column: a 4 by 4 grid with 6 marked cells.
Cross-attention adds a second grid. With a 6 token source and 4 target positions, each decoder position reads all 6 encoder outputs, so each layer performs reads. That is the diagram bridge: two small grids with 24 reads per layer between them.
Why decoder-only won large scale models
Encoder-decoder needs paired data: a sentence plus its translation, a document plus its summary. Paired data is scarce. Decoder-only needs running text: predict the next token everywhere, on any document found anywhere. When the field strategy became training on as much text as possible, the layout that eats unlabeled text won, which is why nearly every large model since is decoder-only even for tasks encoder-decoder used to own.
Code
The point is the mask: build both 4 by 4 grids and confirm 0 against 6 blocked cells, plus the 10 visible ones.
import numpy as np
def causal_mask(n: int) -> np.ndarray: return np.triu(np.ones((n, n)), k=1)
def encoder_only_mask(n: int) -> np.ndarray: return np.zeros((n, n))
n: int = 4enc: np.ndarray = encoder_only_mask(n)dec: np.ndarray = causal_mask(n)print(enc.sum()) # -> 0.0print(dec.sum()) # -> 6.0print(float((np.ones((n, n)) - dec).sum())) # -> 10.0print(bool(dec[0, 1]), bool(enc[0, 1])) # -> True FalsePosition 0 cannot see position 1 under the causal mask and can under the encoder mask, and the visible count is 16 minus 6. The attention math underneath is identical. The mask is the architecture.
Watch Out For
Reading decoder-only as a narrower context
Decoder-only restricts direction, not amount. At position 500 the model sees all of positions 1 through 499 with no restriction. It only cannot peek at 501.
Symptom: claims that decoder-only models understand less context than encoder-only ones, or surprise that long prompt models recall early instructions. Fix: think prefix, not window. Everything behind is visible, everything ahead is not.
Building encoder-decoder out of habit for a generation task
When paired data was the norm this layout was the default. Today a decoder-only model handles most sequence-to-sequence work by framing input as prompt and output as continuation: summarization, translation, rewriting.
Symptom: training a bespoke paired model for a task a prompted large model already does well. Fix: reach for a true encoder-decoder only when you need the encoder unrestricted view of a fixed input beside a separately controlled decoder.
The Quick Version
- One transformer block supports three layouts, split by stack count and whether causal masking applies.
- Encoder-only masks nothing, blocks 0 of 16 cells in a 4 by 4 grid, and builds representations: the BERT lineage for understanding tasks.
- Decoder-only masks the future, blocks 3 plus 2 plus 1, 6 of 16 cells, and generates one token at a time: the GPT lineage.
- Encoder-decoder runs both stacks with cross-attention between them, 24 reads per layer for a 6 token source and 4 targets: the 2017 design for true sequence-to-sequence tasks.
- Decoder-only dominates large models because next token prediction scales to unlabeled text in a way paired data cannot match.