Machine Translation

Machine translation turns a sentence in one language into the same meaning in another, picking each output word from the source words it attends to most.

Three French words become encoder states, attention weights 0.665, 0.245 and 0.090 blend them into one context vector, and the decoder writes the next English word from it.

Why Does This Exist?

Human translators can't keep up with the daily volume of product pages, support tickets, docs and messages that need converting. Rule systems tried to cover each language pair with hand-written grammar rules and dictionaries, and they broke on idioms, different word orders and words with several meanings.

Neural machine translation replaces those rules with a learned map from one sequence to another. It keeps meaning across different lengths and orders, which is why the same idea now powers cross-lingual search and multilingual assistants.

This page covers the encoder-decoder with attention: how source words become states, how the decoder picks what to read, and how output words are chosen one at a time. For turning raw strings into model-ready pieces first, see tokenization in NLP. For scoring whole-sentence meaning instead of translating, see sentence embeddings. For answering from text instead of converting languages, see question answering.

The one idea you need first

Text is split into tokens such as words or subwords, and each token becomes a vector the network can compute with. That split plus the vector is the whole prerequisite. The full treatment is in tokenization in NLP.

Think of It Like This

A bilingual secretary with sticky notes

You dictate a French memo and your secretary writes each phrase on a sticky note, one note per phrase, and spreads them on a desk. Before typing each English word they glance back at the notes and press a finger most firmly on the note that matters for that word.

The notes are the encoder states. The finger pressure is the attention weight: most on one note, some on the rest, always summing to full pressure. The typed English word is the decoder output.

Where it stops: the secretary truly understands both languages and remembers yesterday. The model keeps no meaning outside its vectors, and it forgets everything beyond the weights and the current states.

How It Actually Works

Call the source tokens $X = (x_1, x_2, \ldots, x_n)$ , where each $x_i$ is one source piece such as a word. An embedding layer turns each piece into a vector $e_i$ , a list of numbers that stands for that piece. The encoder reads $(e_1, \ldots, e_n)$ and returns one state per position, $H = (h_1, \ldots, h_n)$ , where each $h_i$ mixes word $i$ with its surrounding context.

At output step $t$ the decoder holds a state $s_t$ , a summary of the English words written so far plus what it has read. It scores every source position against $s_t$ , producing raw scores $z_{t,i}$ , one number per source word. Attention turns those scores into weights $\alpha_{t,i}$ that sum to 1, so $\alpha_{t,i}$ reads as the share of focus on source word $i$ while writing target word $t$ . The context vector $c_t$ is that weighted blend:

c_t = \sum_{i=1}^{n} \alpha_{t,i} h_i

In plain words: $c_t$ is the average of the encoder states, pulled toward whichever source words the weights favour. The decoder then combines $s_t$ with $c_t$ , passes the result through a feed-forward layer and a softmax, and outputs $P(y_t \mid y_{<t}, X)$ , the probability of each candidate next English word $y_t$ given the earlier English words $y_{<t}$ and the full French input $X$ . The chosen word is fed back as input for step $t+1$ , until a stop token ends the sentence.

Why the weights sum to 1

Softmax turns raw scores into shares. With scores $[2, 1, 0]$ , exponentiate to $[7.389, 2.718, 1.000]$ , sum to $11.107$ , and divide to get $[0.665, 0.245, 0.090]$ . They sum to $1.000$ up to rounding, so raising one share must lower the others. That fixed budget is what forces the model to choose.

Worked example

Take the French input "Le chat dort" with three encoder states $h_1 = [1.0, 0.0]$ , $h_2 = [0.0, 1.0]$ , $h_3 = [1.0, 1.0]$ . Suppose the decoder scores for the next English word are $[2.0, 1.0, 0.0]$ , so the weights are $[0.665, 0.245, 0.090]$ as computed above.

The context vector is:

c = 0.665 \times [1.0, 0.0] + 0.245 \times [0.0, 1.0] + 0.090 \times [1.0, 1.0] = [0.755, 0.335]

First coordinate: $0.665 + 0.090 = 0.755$ . Second: $0.245 + 0.090 = 0.335$ . The blend leans toward $h_1$ because its weight is largest, which is exactly the picture above: the thickest connector lands on the first source word, and the context dot sits closest to that state.

Code

This snippet recomputes the weights and the blend from the worked example, so you can check the arithmetic.

1import math2
3
4def softmax(xs: list[float]) -> list[float]:5    m = max(xs)6    exps = [math.exp(x - m) for x in xs]7    s = sum(exps)8    return [e / s for e in exps]9
10
11scores = [2.0, 1.0, 0.0]12w = softmax(scores)13print([round(v, 3) for v in w])  # -> [0.665, 0.245, 0.09]14c0 = w[0] * 1.0 + w[2] * 1.015c1 = w[1] * 1.0 + w[2] * 1.016print(round(c0, 3), round(c1, 3))  # -> 0.755 0.335

Watch Out For

Trusting short outputs on long inputs

Early encoder-decoder models without attention squeezed every sentence into one fixed vector. Short sentences translated fine, then quality fell apart past roughly twenty words because detail had nowhere to live. The symptom is fluent beginnings that drift or drop clauses near the end. The fix is attention or a Transformer that keeps all states available, so the decoder rereads the source at each step instead of memorising it once.

Reading fluency as correctness

A neural translator can output smooth English that adds facts the French never stated, especially names, numbers and negations. The symptom is a confident sentence with zero awkwardness that back-translation exposes as wrong. The fix is to check entities and negation against the source for anything high stakes, and to keep beam outputs or attention peaks around to see which source words each English word actually came from.

The Quick Version

Machine translation maps a source sequence to a target sequence of possibly different length, one output word at a time.
The encoder turns each source token plus context into a state; the decoder writes from those states, not from a single squeezed vector.
Attention weights sum to 1 and blend the states into a context vector; scores [2, 1, 0] give [0.665, 0.245, 0.090].
The worked blend of [1.0, 0.0], [0.0, 1.0] and [1.0, 1.0] under those weights is [0.755, 0.335].
Without attention, long sentences lose detail; with it, each output word rereads the source.
Fluent output can still invent facts, so verify names, numbers and negation against the source.