Seq2Seq with Attention Mechanism

Dynamic alignment that frees encoder-decoder models from fixed-size context vectors. At each decoding step, the model computes a weighted average over all source representations.

Seq2Seq attention enables the decoder to query all encoder states dynamically, constructing a customized context vector for each generated output token.

Why Does This Exist?

The original sequence-to-sequence (Seq2Seq) architecture introduced by Sutskever et al. (2014) and Cho et al. (2014) revolutionized neural machine translation and speech recognition. A recurrent encoder processed an input sequence of variable length $T_x$ , updating its hidden state iteratively until the final token was reached. The final hidden state $h_{T_x}$ served as the sole context vector $c$ , handed over to initialize the decoder.

This design created a crippling mathematical flaw: the fixed-length vector bottleneck. Compressing a complex 40-word legal document or a multi-clause compound sentence into a single 512-dimensional vector forced the network to overwrite early details with later tokens. In practice, translation quality measured by BLEU score collapsed precipitously as sentence lengths exceeded 20 words.

Attention mechanisms, pioneered by Bahdanau et al. in 2014 and refined by Luong et al. in 2015, dissolved this bottleneck entirely. Instead of discarding intermediate encoder representations, the decoder retains direct random access to all hidden states $h_1, \dots, h_{T_x}$ . At every step $t$ of target generation, the decoder constructs a bespoke, dynamic context vector $c_t$ by computing a softmax alignment distribution across the entire source sequence. This breakthrough served as the direct intellectual ancestor of modern self-attention in transformers.

Think of It Like This

A simultaneous interpreter consulting a full written transcript

Imagine a conference interpreter translating a technical keynote speech from German into English.

In the classic fixed-vector Seq2Seq setup, the interpreter is forced to listen to a full three-minute paragraph in complete silence, close their eyes, memorize the entire passage as a single mental impression, and then begin speaking English from raw memory. Early German verbs and noun clauses get garbled or forgotten, because human working memory cannot store forty words of nuanced syntax in a single mental snapshot.

With an attention mechanism, the interpreter works with a live, real-time written stenograph transcript of the German speech resting on their desk. As they speak each English sentence, their eyes actively glance back to the exact German phrase currently being rendered: when translating the German compound adjective, their gaze zooms in on words four and five; when translating the sentence-final separable German verb prefix, their gaze darts to the very end of the paragraph. They generate an output word while dynamically referencing the specific source words that justify it.

How It Actually Works

Additive Versus Multiplicative Alignment Formulations

An attention-equipped sequence-to-sequence model consists of an encoder producing a sequence of hidden state vectors $h = (h_1, h_2, \dots, h_{T_x})$ , and a decoder generating target tokens $y_1, y_2, \dots, y_{T_y}$ using decoder hidden states $s_1, s_2, \dots, s_{T_y}$ .

At each target decoding step $t$ , the model computes a dynamic context vector $c_t$ as the weighted sum of all encoder states:

c_t = \sum_{i=1}^{T_x} \alpha_{t, i} h_i

where the scalar attention weights $\alpha_{t, i}$ form a probability distribution derived from a softmax over scalar alignment scores $e_{t, i}$ :

\alpha_{t, i} = \frac{\exp(e_{t, i})}{\sum_{k=1}^{T_x} \exp(e_{t, k})}

The alignment score $e_{t, i}$ measures how well source position $i$ matches the decoding query state. Two major formulations govern this scoring:

1. Bahdanau Attention (Additive Attention, 2014)

Bahdanau et al. formulate alignment using a single-layer feed-forward network with a non-linear $\tanh$ activation. Scoring evaluates the previous decoder state $s_{t-1}$ against encoder state $h_i$ :

e_{t, i} = v_a^\top \tanh(W_a s_{t-1} + U_a h_i)

where $W_a \in \mathbb{R}^{d_a \times d_{\text{dec}}}$ , $U_a \in \mathbb{R}^{d_a \times d_{\text{enc}}}$ , and $v_a \in \mathbb{R}^{d_a}$ are trainable projection matrices. The context vector $c_t$ is computed before the current decoder hidden state $s_t$ is updated:

s_t = f(s_{t-1}, y_{t-1}, c_t)

2. Luong Attention (Multiplicative Attention, 2015)

Luong et al. simplify the alignment calculation by utilizing the current decoder hidden state $s_t$ , and propose three distinct scoring functions:

Dot: $e_{t, i} = s_t^\top h_i$ (requires $d_{\text{dec}} = d_{\text{enc}}$ )
General (Multiplicative): $e_{t, i} = s_t^\top W_a h_i$
Concat: $e_{t, i} = v_a^\top \tanh(W_a [s_t \,;\, h_i])$

Once $c_t$ is assembled, Luong combines the context vector and current decoder state through an attentional hidden state layer $\tilde{s}_t$ :

\tilde{s}_t = \tanh(W_c [c_t \,;\, s_t])

The predictive probability for the next token is then projected directly: $p(y_t \mid y_{<t}, x) = \text{softmax}(W_s \tilde{s}_t)$ .

Worked Example

Suppose an encoder processes a 3-word source sentence ( $T_x = 3$ ) with 2-dimensional hidden states $h_1, h_2, h_3 \in \mathbb{R}^2$ :

h_1 = \begin{bmatrix} 1.00 \\ 0.00 \end{bmatrix} \quad (\text{"The"}), \quad h_2 = \begin{bmatrix} 0.00 \\ 2.00 \end{bmatrix} \quad (\text{"black"}), \quad h_3 = \begin{bmatrix} 1.00 \\ 1.00 \end{bmatrix} \quad (\text{"cat"})

The decoder is currently generating target token $y_t =$ "noire" (French feminine adjective for "black"). Its query state is:

s_t = \begin{bmatrix} 0.20 \\ 1.50 \end{bmatrix}

Using Luong Dot scoring ( $e_{t, i} = s_t^\top h_i$ ):

Step 1: Compute Raw Alignment Scores

e_{t, 1} = s_t^\top h_1 = (0.20 \times 1.00) + (1.50 \times 0.00) = 0.20

e_{t, 2} = s_t^\top h_2 = (0.20 \times 0.00) + (1.50 \times 2.00) = 3.00

e_{t, 3} = s_t^\top h_3 = (0.20 \times 1.00) + (1.50 \times 1.00) = 0.20 + 1.50 = 1.70

Step 2: Evaluate Softmax Probabilities $\alpha_{t, i}$

Compute exponentials:

e^{0.20} \approx 1.2214, \quad e^{3.00} \approx 20.0855, \quad e^{1.70} \approx 5.4739

Sum of exponentials:

\sum_{k=1}^3 e^{e_{t, k}} = 1.2214 + 20.0855 + 5.4739 = 26.7808

Calculate normalized weights:

\alpha_{t, 1} = \frac{1.2214}{26.7808} \approx 0.0456 \quad (4.56\%)

\alpha_{t, 2} = \frac{20.0855}{26.7808} \approx 0.7500 \quad (75.00\%)

\alpha_{t, 3} = \frac{5.4739}{26.7808} \approx 0.2044 \quad (20.44\%)

Step 3: Compute the Dynamic Context Vector $c_t$

c_t = 0.0456 \begin{bmatrix} 1.00 \\ 0.00 \end{bmatrix} + 0.7500 \begin{bmatrix} 0.00 \\ 2.00 \end{bmatrix} + 0.2044 \begin{bmatrix} 1.00 \\ 1.00 \end{bmatrix}

c_{t, 1} = (0.0456 \times 1.00) + (0.7500 \times 0.00) + (0.2044 \times 1.00) = 0.0456 + 0.2044 = 0.2500

c_{t, 2} = (0.0456 \times 0.00) + (0.7500 \times 2.00) + (0.2044 \times 1.00) = 1.5000 + 0.2044 = 1.7044

c_t = \begin{bmatrix} 0.2500 \\ 1.7044 \end{bmatrix}

Because $75\%$ of the attention mass centers on $h_2$ , $c_t$ carries the semantic information of "black", providing the exact features required to output the feminine adjective form "noire".

Code

1import numpy as np2
3def compute_seq2seq_attention(4    encoder_states: np.ndarray,5    decoder_query: np.ndarray,6    method: str = "dot",7    W_general: np.ndarray | None = None8) -> tuple[np.ndarray, np.ndarray]:9    """Compute attention alignment distribution and dynamic context vector."""10    # encoder_states: (T_x, d), decoder_query: (d,)11    if method == "dot":12        scores = np.dot(encoder_states, decoder_query)13    elif method == "general":14        if W_general is None:15            raise ValueError("W_general matrix required for general attention")16        # s_t^T * W * h_i17        transformed = np.dot(encoder_states, W_general.T)18        scores = np.dot(transformed, decoder_query)19    else:20        raise ValueError(f"Unsupported scoring method: {method}")21
22    # Numerically stable softmax23    shift_scores = scores - np.max(scores)24    exp_scores = np.exp(shift_scores)25    alpha = exp_scores / np.sum(exp_scores)26
27    # Dynamic context vector: c_t = sum_i alpha_i * h_i28    context_vector = np.sum(alpha[:, np.newaxis] * encoder_states, axis=0)29
30    return np.round(context_vector, 4), np.round(alpha, 4)31
32# 3 encoder hidden states (dimension d = 2)33h_encoder = np.array([34    [1.00, 0.00],  # "The"35    [0.00, 2.00],  # "black"36    [1.00, 1.00],  # "cat"37], dtype=np.float64)38
39# Current decoder state querying for "noire"40s_decoder = np.array([0.20, 1.50], dtype=np.float64)41
42c_t, alpha_weights = compute_seq2seq_attention(h_encoder, s_decoder, method="dot")43
44print(f"Alignment Weights (alpha): {alpha_weights.tolist()}")45# -> Alignment Weights (alpha): [0.0456, 0.75, 0.2044]46
47print(f"Context Vector (c_t): {c_t.tolist()}")48# -> Context Vector (c_t): [0.25, 1.7044]

Watch Out For

Computing quadratic cross-attention matrices without key-query dimension scaling causes gradient saturation

In dot-product attention scoring $e_{t, i} = s_t^\top h_i$ , when hidden state dimension $d$ grows large (e.g., $d \ge 512$ ), the variance of the dot product scales linearly with $d$ : $\operatorname{Var}(s_t^\top h_i) = d$ .

This causes raw alignment scores to explode in magnitude, pushing the subsequent softmax function into saturated regions where local gradients approach zero ( $\alpha_i(1 - \alpha_i) \approx 0$ ). In deep architectures, this prevents backpropagation from updating alignment projections. Vaswani et al. solved this by scaling scores by $\frac{1}{\sqrt{d_k}}$ , a critical practice whenever deploying dot-product attention in high dimensions.

The Quick Version

Attention replaces the fixed single-vector encoder bottleneck with a dynamically calculated, step-specific context vector $c_t$ .
Alignment scores evaluate compatibility between decoder query states and all source encoder hidden states.
Bahdanau computes additive attention via non-linear projection, while Luong utilizes fast multiplicative dot products.