Attention Mechanism

Instead of reading one fixed summary vector, let the decoder look back at every encoder state and weigh each one by how relevant it is to the current step.

Scores [2,1,0] soften into weights [0.665,0.245,0.090] that blend three encoder states into one context vector

Why Does This Exist?

Start with sequence-to-sequence: an encoder squeezes a whole sentence into one fixed-size vector, and a decoder must write the translation from that vector alone. Short sentences survive this squeeze. Long ones don't. Quality visibly falls as sentences grow, because a fixed container cannot hold an unbounded sentence, no matter how well you train.

Picture translating a 40-word contract clause. The decoder opens strongly while the summary is fresh, then drops a clause halfway through and repeats a phrase near the end. It isn't confused about grammar. It has lost the source. Researchers hit exactly this wall in 2014 and 2015, and the fix is this page: stop summarizing the sentence once, and let the decoder look back at every encoder state at every step instead.

This page covers that original fix, the seq2seq-era attention behind Bahdanau-style translation: score each encoder state, soften the scores into weights, blend by those weights. Its children each get their own page. Self-attention drops the encoder and decoder split and lets a sequence read itself. Cross-attention rebuilds the same idea on query-key-value machinery inside transformers. Transformers then throw away recurrence entirely. If you haven't met the bottleneck yet, read sequence-to-sequence first.

Think of It Like This

The translator who keeps the letter open

You're translating a long letter with the source face-down on the desk, and you may glance at it only once before you start. A short note survives. A four-page letter doesn't: by page two you're guessing. Now someone lets you keep the letter open beside you. For each sentence you write, you glance at the paragraph that matters for that sentence, linger where it's dense, skim where it's not, and move on. Each glance differs because each sentence needs something different.

The score is how long your eyes rest on each paragraph. The weights are those linger times turned into shares of your attention. The blend is the sentence you write from that glance. Nothing was ever memorized whole.

Where it stops: a glance can't add what the letter never said. If the encoder misread a word, attention faithfully attends to the misreading. It routes information, it doesn't repair it.

How It Actually Works

Score, weigh, blend

At each decoder step $t$ , attention runs three operations over every encoder hidden state $e_1, \ldots, e_n$ . Here $e_i$ is the encoder's reading of source position $i$ , and $d_t$ is the decoder's current state asking what it needs right now:

Score. Compare $d_t$ against each $e_i$ for one number per position. One common form is a learned bilinear score with a weight matrix $W_a$ :

\text{score}(d_t, e_i) = d_t^\top W_a e_i

Additive attention runs the pair through a small feedforward layer instead. The scoring function varies; everything after it stays the same.

Weigh. Soften the $n$ raw scores into shares with softmax, where $\alpha_{t,i}$ (read "alpha") is the weight on position $i$ at step $t$ :

\alpha_{t,i} = \frac{\exp(\text{score}(d_t, e_i))}{\sum_{j=1}^{n} \exp(\text{score}(d_t, e_j))}

Each $\alpha_{t,i}$ sits between 0 and 1, and the row sums to exactly 1: a genuine distribution over which source position matters right now.

Blend. Mix the encoder states by those shares into this step's context vector $c_t$ :

c_t = \sum_{i=1}^{n} \alpha_{t,i} \, e_i

The decoder reads $c_t$ alongside $d_t$ to emit the next word. Next step brings a new $d_{t+1}$ , fresh scores, fresh weights, fresh $c_{t+1}$ . Nothing is cached across steps, because what matters moves as the sentence grows.

Worked example

Say one decoder step scores three encoder states $[2, 1, 0]$ . Exponentiate: $e^2 \approx 7.39$ , $e^1 \approx 2.72$ , $e^0 = 1.00$ . Add them: $7.39 + 2.72 + 1.00 = 11.11$ . Divide each by the sum:

\alpha = [7.39 / 11.11, \; 2.72 / 11.11, \; 1.00 / 11.11] \approx [0.665, 0.245, 0.090]

The shares sum to $1.000$ . Notice the top score earns two thirds of the weight, not all of it: softmax exaggerates gaps but never crowns a winner, which is why attention blends instead of selecting.

Now give those three states values $[4.0, 2.0, 0.0]$ , small stand-ins for encoder readings. The context is their weighted mix:

c = 0.665 \times 4.0 + 0.245 \times 2.0 + 0.090 \times 0.0 = 2.66 + 0.49 + 0.00 = 3.15

One number out, $3.15$ , leaning hard toward the first state yet still touched by the second. That's the diagram above: score row, normalized weights, blended vector.

Code

Same step, run for real: soften the scores, check they sum to one, mix the values.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e: np.ndarray = np.exp(z - np.max(z))6    return e / np.sum(e)7
8
9scores: np.ndarray = np.array([2.0, 1.0, 0.0])10weights: np.ndarray = softmax(scores)11values: np.ndarray = np.array([4.0, 2.0, 0.0])12context: float = float(weights @ values)13
14print(f"{weights[0]:.3f} {weights[1]:.3f} {weights[2]:.3f}")  # -> 0.665 0.245 0.09015print(f"{float(np.sum(weights)):.1f}")  # -> 1.016print(f"{context:.2f}")  # -> 3.15

The printout matches the hand computation: weights $0.665$ , $0.245$ , $0.090$ summing to $1.0$ , and a blended context of $3.15$ .

Watch Out For

Reusing one step's weights for the next step

Attention weights carry a step index for a reason: $\alpha_{t,i}$ answers what matters while writing word $t$ . Freeze our $[0.665, 0.245, 0.090]$ and reuse it for the following word, and the decoder keeps staring at position one after the sentence has moved on. Early words look fine and later words drift, drop clauses, or repeat, which reads like a training problem but is really a plumbing one.

Symptom: translations start strong and decay, with later words echoing earlier alignments. Fix: recompute score, softmax, and blend fresh at every decoder step, and never cache $\alpha$ across steps.

Letting raw dot products sharpen the softmax into a spike

Dot products grow with hidden size, and softmax turns large spread-out scores into a near one-hot spike: nearly all weight on one position with gradients near zero everywhere else. Our soft $[0.665, 0.245, 0.090]$ would harden toward $[0.99, 0.01, 0.00]$ before the model has learned anything worth pointing at.

Symptom: weights sit near one-hot from the first epoch while training stalls. Fix: divide scores by $\sqrt{d_k}$ , where $d_k$ is the key dimension, before the softmax, which is the scaled form modern attention uses by default.

The Quick Version

Attention retires the one-vector summary: at every step the decoder scores all encoder states, softens the scores into weights, and blends by them.
Scores $[2, 1, 0]$ become weights $[0.665, 0.245, 0.090]$ , which sum to $1.000$ and mix values $[4.0, 2.0, 0.0]$ into a context of $3.15$ .
The winner takes two thirds, never everything, so the context stays a blend with gradients flowing everywhere.
Fresh weights every step, because the relevant source position moves as the output grows.
When weights freeze near one-hot before learning anything, suspect unscaled scores and divide by $\sqrt{d_k}$ .