Core ML

Cross-Attention

Take queries from one sequence and keys and values from a different sequence entirely, so one sequence can read another without the two ever being merged into one.

Queries come from the decoder's own sequence while keys and values come from the encoder's output, so the decoder reads a different sequence than the one producing its queries

Why Does This Exist?

Self-attention fixes queries, keys, and values to the same sequence — every token attends over the sequence it belongs to. That's exactly right for building a representation of a single passage of text. It's exactly wrong for a task like translation, where a decoder generating French words needs to look back at an English source sentence that is a genuinely different sequence, with a different length, existing for a different reason.

The original attention mechanism that fixed the sequence-to-sequence bottleneck was already doing this — a decoder's hidden state scoring against an encoder's hidden states. Cross-attention is that same idea, rebuilt on the query-key-value machinery self-attention introduced, so it plugs into the transformer block architecture directly instead of needing its own bespoke scoring function.

Think of It Like This

Consulting a reference document while writing your own

Self-attention is rereading your own draft to keep it internally consistent — checking that paragraph three doesn't contradict paragraph one. Cross-attention is closing your draft and consulting a separate reference document while you write: a source article you're summarizing, a table of data you're describing, a photograph you're captioning.

The document you're producing and the document you're consulting are two different objects. You're not re-reading your own words when you check the reference; you're asking it a question — "what does the source say happened at this point?" — and pulling back whatever answers that question. Your query comes from what you're currently writing. The reference material supplies everything you might retrieve.

How It Actually Works

Two sequences, one attention computation

Cross-attention uses the exact same formula as self-attention:

\text{CrossAttention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V

The only difference — and it's the entire difference — is where $Q$ , $K$ , and $V$ come from. In self-attention, all three are projections of the same sequence. In cross-attention, $Q$ is a projection of one sequence (the decoder's current representation, in the diagram above), while $K$ and $V$ are projections of a different sequence (the encoder's output). The two sequences can even have different lengths: if the source has $n$ tokens and the target has $m$ , the resulting score matrix $QK^\top$ is $m \times n$ , not square — every target position scores against every source position, but there's no requirement that the counts match.

What this buys architecturally

Because $K$ and $V$ come from a sequence that's already been fully computed, cross-attention lets a decoder condition its generation on that entire other sequence without ever mixing the two representations together into one shared space. The encoder's output stays exactly what the encoder produced; the decoder just reads it, the same way you'd read a reference document without editing it in place. This separation is what makes cross-attention the general-purpose "condition generation on X" mechanism — X doesn't have to be text at all.

Beyond translation

The reason cross-attention outlived the original translation architecture it appeared in is that "condition one sequence on another" turns out to be an enormously general pattern. A vision-language model uses cross-attention to let text generation condition on image patch embeddings — the query comes from the text being generated, the keys and values come from the visual encoder. A diffusion model conditions image generation on a text prompt's embeddings the same way. Any time a model needs to generate one thing while reading another, cross-attention is the mechanism that connects the two without forcing them into a shared representation from the start.

Show Me the Code

The same attention function as self-attention, but with $Q$ built from one input and $K$ , $V$ from another — the entire distinction made explicit in the function signature.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max(axis=-1, keepdims=True))6    return e / e.sum(axis=-1, keepdims=True)7
8
9def cross_attention(10    decoder_seq: np.ndarray, encoder_seq: np.ndarray, w_q: np.ndarray, w_k: np.ndarray, w_v: np.ndarray11) -> np.ndarray:12    q = decoder_seq @ w_q          # queries from the DECODER sequence13    k = encoder_seq @ w_k          # keys from the ENCODER sequence14    v = encoder_seq @ w_v          # values from the ENCODER sequence too15    scores = q @ k.T / np.sqrt(k.shape[-1])   # shape (m, n) — need not be square16    return softmax(scores) @ v17
18
19rng = np.random.default_rng(0)20decoder_tokens = rng.normal(size=(3, 8))   # 3 target-side positions21encoder_tokens = rng.normal(size=(5, 8))   # 5 source-side positions — different length22w_q, w_k, w_v = (rng.normal(scale=0.3, size=(8, 4)) for _ in range(3))23out = cross_attention(decoder_tokens, encoder_tokens, w_q, w_k, w_v)24print(out.shape)  # -> (3, 4) — one output per decoder position, regardless of source length

The output has one row per decoder token no matter how long the encoder sequence is — the source length only affects how many keys get scored against, not the shape of what comes back.

Watch Out For

Confusing cross-attention with concatenating the two sequences

A tempting shortcut is to just concatenate the two sequences and run ordinary self-attention over the combined result, letting every position attend to every other position regardless of which sequence it came from. That's a real, different architecture with real consequences: it lets decoder positions attend to other decoder positions directly through this pathway too, and it lets encoder positions attend to decoder positions, which usually breaks the intended information flow (the encoder representation should stay fixed, independent of what the decoder is doing). Cross-attention's query/key-value split deliberately keeps that flow one-directional.

Forgetting cross-attention still needs its own learned projections

Because cross-attention reuses self-attention's formula, it's easy to assume the same $W_Q$ , $W_K$ , $W_V$ weights could be shared between a model's self-attention and cross-attention layers. They can't, and frameworks don't share them — self-attention's projections are learned to relate a sequence to itself, cross-attention's are learned to relate one sequence to a structurally different one, and conflating them (or accidentally wiring the wrong weight matrix in a custom implementation) trains a model that can't do either job well.

The Quick Version

Cross-attention uses the same score-softmax-blend formula as self-attention, but $Q$ comes from one sequence while $K$ and $V$ come from a different one.
The two sequences can have different lengths — the resulting score matrix doesn't need to be square.
It lets one sequence condition on another without merging their representations into a shared space.
It generalizes well beyond translation: vision-language fusion and text-conditioned diffusion both use the same mechanism.
Self-attention and cross-attention layers need their own separate learned projections — they aren't interchangeable.