Core ML

Encoder-Decoder Architectures

Three ways to arrange the very same transformer block: read the whole input at once, generate one token at a time, or bridge both with cross-attention.

Three transformer variants built from the same block: encoder-only stacks with no causal mask, decoder-only stacks with causal masking, and encoder-decoder running both with cross-attention between them

Why Does This Exist?

The transformer architecture defines one repeatable block: attention, then a feed-forward network, each wrapped in a residual connection. But a block is not a model — you still have to decide how many stacks of it to build, whether each stack can look ahead at tokens it hasn't reached yet, and whether one stack gets to read another's output. Those three decisions aren't tweaks; they produce three genuinely different machines from the identical building block.

The original 2017 transformer paper answered all three questions one way, for machine translation: two stacks, an encoder that reads the whole source sentence with no restriction, and a decoder that generates the target sentence one word at a time while reading the encoder's output. That's an encoder-decoder model. But nothing forces you to build both stacks. Drop the decoder and you get an encoder-only model, built to understand a complete input rather than generate anything — that's BERT's lineage. Drop the encoder and keep only the causally-masked decoder, and you get a decoder-only model, built to generate text one token after another — that's GPT's lineage. Same block, three skeletons, three different jobs.

Think of It Like This

One factory floor, three different assembly plans

Picture one factory floor stocked with identical assembly stations — the transformer block. A plant manager can build one production line where every station sees the entire order sheet before starting (encoder-only): good for grading and sorting a finished order, useless for building one from scratch. A different manager builds a line where each station only sees what's been assembled so far, never the finished product (decoder-only): perfect for constructing something step by step. A third manager builds both lines and adds a conveyor belt so the generating line can check back against the finished order sheet at every step (encoder-decoder): the right setup when the job is genuinely "turn this input into a different output," like translation.

Same stations everywhere. What differs is wiring: how many lines, whether a station can see ahead, and whether one line checks another's work.

How It Actually Works

Encoder-only: no restriction, built to understand

An encoder-only model stacks transformer blocks with no causal masking at all — every position attends freely to every other position in the input, both before and after it. That's the right design when the job is building one good representation of a complete input: classifying a sentence's sentiment, tagging named entities, scoring whether two sentences mean the same thing. There's no generation step, so there's no reason to hide any part of the input from any position. BERT is the canonical encoder-only model, and it's still the right tool whenever you need a representation rather than generated text.

Decoder-only: causal masking throughout, built to generate

A decoder-only model stacks blocks where every position is causally masked — it can attend to itself and everything before it, never anything after. That restriction is exactly what makes autoregressive generation coherent: train the model to predict the next token everywhere in a sequence, and at inference time you can run it one step at a time, always extending, never needing to see the ending in advance because it was never allowed to during training either. GPT is the canonical decoder-only model.

Encoder-decoder: both stacks, bridged by cross-attention

An encoder-decoder model runs an unmasked encoder stack over the source input and a causally-masked decoder stack that generates the output, with the decoder additionally running cross-attention against the encoder's output at every layer — queries from the decoder, keys and values from the encoder. That's the architecture the original transformer paper introduced, and it's still the natural fit for genuine sequence-to-sequence tasks where the output is a transformation of a specific input: translation, summarization, and T5-style models that frame many tasks as text-to-text.

Why decoder-only won for large language models

Encoder-decoder needs paired input-output data — a source sentence and its translation, a document and its summary. That kind of labeled pairing is comparatively scarce. Decoder-only needs nothing but running text: predict the next token, everywhere, on any document you can find. That single objective scales to effectively unlimited unlabeled text in a way paired data never could. When the field's dominant strategy became "train on as much text as possible," decoder-only was built for exactly that, and it's why almost every large-scale LLM since is decoder-only, even for tasks encoder-decoder models used to own.

Show Me the Code

A toy comparison of what each variant's attention mask actually allows — full visibility, causal-only, or a decoder cross-attending into an encoder.

1import numpy as np2
3
4def causal_mask(n: int) -> np.ndarray:5    return np.triu(np.ones((n, n)), k=1)          # 1s strictly above the diagonal6
7
8def encoder_only_mask(n: int) -> np.ndarray:9    return np.zeros((n, n))                        # nothing blocked — full visibility10
11
12rng = np.random.default_rng(0)13n = 414enc_mask = encoder_only_mask(n)15dec_mask = causal_mask(n)16print(enc_mask.sum())          # -> 0.0 — no position is ever blocked17print(dec_mask.sum())          # -> 6.0 — 6 forward-looking positions blocked in a 4x4 grid18print(bool(dec_mask[0, 1]))    # -> True — position 0 cannot see position 119print(bool(enc_mask[0, 1]))    # -> False — the encoder has no such restriction

The mask is the entire architectural difference between "understand everything at once" and "generate one step at a time" — same attention computation underneath, different matrix of what's allowed.

Watch Out For

Assuming decoder-only means the model can never see bidirectional context

Decoder-only means each generation step can't see ahead of itself — it says nothing about how much context came before. A decoder-only model at position 500 has full, unrestricted access to everything from position 1 to 499; it just can't peek at 501. People sometimes describe decoder-only models as having a fundamentally narrower context than encoder-only ones, which isn't accurate — the restriction is about direction, not about how much prior context is visible.

Reaching for an encoder-decoder model out of habit for a generation task

Encoder-decoder made sense when paired data was the norm and decoder-only LLMs weren't yet dominant. Today, a decoder-only model handles most sequence-to-sequence tasks fine by framing the input as a prompt and the output as a continuation — summarization, translation, and rewriting all run through decoder-only LLMs routinely. Reach for a true encoder-decoder design only when you specifically need the encoder's unrestricted view of a fixed input alongside a separately-controllable decoder.

The Quick Version

The same transformer block supports three architectures, distinguished by how many stacks exist and whether causal masking applies.
Encoder-only has no masking and builds a representation of a complete input — BERT's lineage, good for understanding tasks.
Decoder-only causally masks every position and generates one token at a time — GPT's lineage, good for generation.
Encoder-decoder runs both stacks, with the decoder cross-attending into the encoder's output — the original transformer's design, good for true sequence-to-sequence tasks.
Decoder-only became the dominant large-model choice because next-token prediction scales to unlabeled text in a way paired data can't match.