Cross-Attention

One decoder token scores its query against every encoder key, then blends encoder values by those weights, so generation reads a different sequence without merging the two.

One decoder query scores three encoder keys, softens to weights that sum to one, and blends three encoder values into one output

Why Does This Exist?

Self-attention pins queries, keys, and values to the same sequence. Every token reads the sequence it belongs to. That is exactly right when you build a picture of one passage. It is the wrong tool when a French decoder must read an English source sentence, a genuinely different sequence with its own length and its own job.

The older fix for this sat inside sequence-to-sequence models. An encoder squeezed a whole sentence into one fixed vector, and quality fell as sentences grew because one container cannot hold an unbounded sentence. The attention mechanism stopped the single squeeze and let the decoder score every encoder state at every step. Cross-attention is that same idea rebuilt on query-key-value machinery, so it slots straight into transformer blocks.

This page covers the split-source version: where each matrix comes from, how scaling and softmax turn scores into shares, and one hand-worked blend you can check. Self-attention is the sibling contrast with all three matrices from one sequence. Attention mechanism is the prerequisite score, weigh, and blend idea. Transformers show the full stack where cross-attention lives. If the bottleneck is new to you, read sequence-to-sequence first.

Think of It Like This

Describing a photo while you write

You write a caption with the photo propped beside the page. For each word you glance at the patch that word needs, linger where the detail sits, skim the rest, and write. Each glance differs because each word asks something different.

Your half-written line is the query, each photo patch is a key plus the detail it holds, your linger time is the weight, and the word you write is the blend. The page and the photo never merge into one object.

Where it stops: looking cannot add what the photo lacks. If the lens blurred a street sign, attention faithfully reads the blur. It routes the right patch to the right word, it does not repair the picture.

How It Actually Works

Cross-attention runs the familiar attention math with an unfamiliar split. The query asks from the target side. The keys and values answer from the source side.

Where do Q, K, and V come from?

Call the decoder states $X_{dec}$ , with $m$ rows for $m$ target positions, and the encoder outputs $X_{enc}$ , with $n$ rows for $n$ source positions. Here a row means one token vector, and $m$ and $n$ rarely match in translation. Three learned matrices $W_Q$ , $W_K$ , and $W_V$ project both sides into one shared width $d_k$ , where $d_k$ (read dee-kay) is the key dimension:

Q = X_{dec} W_Q, \qquad K = X_{enc} W_K, \qquad V = X_{enc} W_V

So $Q$ holds $m$ queries from the decoder, $K$ holds $n$ keys from the encoder, and $V$ holds $n$ values from the encoder. A query says what the current target position needs. A key says what a source position offers. A value carries the source content that flows forward when its key matches. Because $K$ and $V$ come from fully computed encoder output, the decoder reads that output without editing it in place.

Score, scale, soften, blend

Every query scores against every key in one matrix multiply, then scales, softens, and blends:

\text{CrossAttention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V

$QK^\top$ is $m$ by $n$ , one row per decoder position and one column per encoder position, so the table need not be square. Dividing by $\sqrt{d_k}$ keeps the scores near unit variance when query and key parts start with zero mean and unit variance, which stops early softmax spikes. Softmax turns each row into weights $\alpha$ that sit between 0 and 1 and sum to exactly 1. Multiplying by $V$ mixes the encoder values by those shares. The diagram above shows this for one row: one query, three keys, three weights, one blended output.

Worked example: one query against three keys

Take $d_k = 4$ , so $\sqrt{d_k} = 2$ , with decoder query $q = [1, 1, 0, 0]$ and three encoder keys $k_1 = [2, 2, 0, 0]$ , $k_2 = [1, 0, 0, 0]$ , and $k_3 = [-1, -1, 0, 0]$ . Raw dot products give:

q \cdot k_1 = 4, \qquad q \cdot k_2 = 1, \qquad q \cdot k_3 = -2

Scale by 2:

[4, 1, -2] / 2 = [2.0, 0.5, -1.0]

Exponentiate for softmax: $e^{2.0} \approx 7.39$ , $e^{0.5} \approx 1.65$ , $e^{-1.0} \approx 0.37$ . Add them: $7.39 + 1.65 + 0.37 = 9.41$ . Divide each by the sum:

\alpha \approx [7.39 / 9.41, \; 1.65 / 9.41, \; 0.37 / 9.41] \approx [0.786, 0.175, 0.039]

The shares sum to $1.000$ . The leader takes most of the mass without taking all of it, which is why the output stays a blend.

Give those positions scalar stand-in values $[4.0, 2.0, 0.0]$ and mix:

c = 0.786 \times 4.0 + 0.175 \times 2.0 + 0.039 \times 0.0 = 3.144 + 0.350 + 0.000 \approx 3.49

One number out, $3.49$ , leaning hard toward the first encoder state yet still touched by the second. The picture draws these same numbers: scaled scores $[2.0, 0.5, -1.0]$ , weights $[0.786, 0.175, 0.039]$ , values $[4.0, 2.0, 0.0]$ , output $3.49$ .

Code

Same step, run for real: soften the scaled scores, check they sum to one, mix the values.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e: np.ndarray = np.exp(z - np.max(z))6    return e / np.sum(e)7
8
9scores: np.ndarray = np.array([2.0, 0.5, -1.0])10weights: np.ndarray = softmax(scores)11values: np.ndarray = np.array([4.0, 2.0, 0.0])12context: float = float(weights @ values)13
14print(f"{weights[0]:.3f} {weights[1]:.3f} {weights[2]:.3f}")  # -> 0.786 0.175 0.03915print(f"{float(np.sum(weights)):.1f}")  # -> 1.016print(f"{context:.2f}")  # -> 3.49

The printout matches the hand computation: weights $0.786$ , $0.175$ , $0.039$ summing to $1.0$ , and a blended context of $3.49$ .

Watch Out For

Letting padding soak up the weights

Source batches pad short sentences to equal length, and those pad positions still score unless you stop them. Our $[0.786, 0.175, 0.039]$ would flatten toward the pads, and the $3.49$ would drift toward whatever vector the pads carry, often zeros that drag every context down.

Symptom: short sentences translate worse than long ones in the same batch, and weight plots show mass sitting on trailing pad columns. Fix: add a pad mask of negative infinity to padded scores before the softmax, so pads get exactly $0$ and the live weights still sum to $1$ .

Building the query from the wrong side

In custom code it is easy to wire $Q$ from the encoder and $K$ , $V$ from the decoder, since the call looks symmetric. The shapes still multiply when lengths differ, but the output then has one row per source token instead of one per target token, and generation conditions on itself while ignoring the source it should read.

Symptom: output length tracks the source length, or loss drops while translations read fluently yet miss source content. Fix: keep the direction fixed, $Q$ always from the querying target sequence and $K$ , $V$ always from the read source sequence, and assert the output has $m$ rows for $m$ decoder positions.

The Quick Version

Cross-attention keeps the score, softmax, and blend formula but takes queries from the decoder while keys and values come from the encoder.
Raw dots [4, 1, -2] scale by 2 into [2.0, 0.5, -1.0], soften into [0.786, 0.175, 0.039], and blend values [4.0, 2.0, 0.0] into 3.49.
The score table spans m target rows by n source columns, so source and target lengths never need to match.
Encoder keys and values get built once and reused, while each new decoder token builds a fresh query.
Padding positions need a mask before the softmax, or the shares leak onto tokens that carry no meaning.