Skip to content
AI360Xpert
Core ML
Visual explainer

Attention Mechanism

Instead of compressing an entire sentence into one vector, attention lets the decoder look back at the full input and blend what matters for the current word.

The classic sequence-to-sequence bottleneck forces the entire input sentence to be compressed into a single, fixed-size context vector.
The classic sequence-to-sequence bottleneck forces the entire input sentence to be compressed into a single, fixed-size context vector.

In early encoder-decoder models, the entire input sentence had to survive being compressed into one fixed-size vector. The decoder was starved for information because there was a hard limit on how much context that single vector could carry.

Score Every State

At each step, the decoder state is compared against every single encoder state to produce a relevance score.
At each step, the decoder state is compared against every single encoder state to produce a relevance score.

Instead of reading one static summary, attention keeps the full input available. At each step, the model computes a dot-product score between the current decoder state and every encoder state, measuring how relevant each input word is right now.

Softmax and Blend

Scores are converted into probabilities using a softmax, then used to compute a weighted sum of the encoder states.
Scores are converted into probabilities using a softmax, then used to compute a weighted sum of the encoder states.

The raw scores are passed through a softmax function, turning them into weights that sum to exactly 1.0. These weights are then used to blend all the encoder states together into a single context vector tailored specifically for this step.

Recomputed Every Step

The context vector isn't fixed once; it is entirely rebuilt with fresh weights for each new word being generated.
The context vector isn't fixed once; it is entirely rebuilt with fresh weights for each new word being generated.

The context vector is never reused. Because the decoder state changes after generating a word, the scores change too. The model dynamically shifts its focus, placing heavy weights on the specific input words that matter most for the next output.

Where It Breaks

Unscaled large dot products push the softmax to a 1-hot spike, collapsing the gradient.
Unscaled large dot products push the softmax to a 1-hot spike, collapsing the gradient.

As the hidden dimension grows, raw dot products get very large. A softmax applied to large, spread-out scores collapses into a near one-hot spike, putting all weight on a single position and killing the gradients everywhere else. This is why modern attention always scales the scores down.

The Quick Version

  • The bottleneck: compressing an entire sentence into one fixed vector.
  • The primitive: scoring the current decoder state against every encoder state.
  • The mechanism: softmax the scores, then blend encoder states by those weights.
  • The payoff: a fresh, dynamic context vector for every output step.
  • The break: unscaled scores cause the softmax to collapse into a spike.

What to Read Next