Machine Translation
Translate a whole sentence by reading its full meaning first, then writing it fresh in the target language. Word order and idioms stop being blockers once the model carries meaning instead of single words.
Why Does This Exist?
A dictionary lookup cannot translate. English writes the adjective before the noun in red car while French writes it after in voiture rouge, so slot-by-slot swapping gives rouge voiture, which is wrong. Idioms break harder. The phrase it is raining cats and dogs has nothing to do with animals, and the word bank means a river edge in one sentence and a money house in the next.
The fix is to carry meaning across the gap instead of words. That is what this page covers: how an encoder reads the full source sentence, how a decoder writes the target sentence word by word, and how attention picks the right source word for each output word. The reusable reader-writer pattern lives in encoder-decoder architectures. The lookup trick that picks source words lives in attention mechanism. The architecture that replaced recurrence with pure attention lives in transformers.
Two basics make the rest click. A token is one word piece the model reads, and an embedding turns that piece into a short list of numbers. A context vector is one fixed list of numbers meant to hold a whole sentence. If you have not seen summaries built this way, start with text summarization, which uses the same reader-writer shape for a different job.
Think of It Like This
An interpreter who waits for the whole sentence
You sit in a booth while a speaker talks. You do not shout a translation after every single word, because the verb may still be coming and the word order may flip. You listen to a full phrase, build a mental picture of what was meant, then say a fresh sentence that carries that picture.
The speaker is the source sentence, your mental picture is the context vector, and your fresh sentence is the decoder output. Listening first lets you flip red car into voiture rouge without thinking twice.
Where it stops: a human interpreter can ask for clarification and knows when a joke will not survive translation. The network always emits numbers, never asks, and has no notion of when to refuse. You must add the rule that catches bad inputs outside the model.
How It Actually Works
Call the source words , meaning the, red and car. Call the encoder states , meaning one vector per source word. Call the decoder state , meaning what has been written so far at step . The encoder reads left to right and folds every word into its state. The decoder starts from that summary and emits one target word at a time, feeding each emitted word back in for the next step.
Early reader-writer models squeezed all of into one context vector . That bottleneck held short phrases but leaked early words on long sentences, which is why a fifty-word sentence lost its beginning. Attention removes the squeeze. At each output step the decoder scores every encoder state, turns the scores into weights that sum to one, and builds a fresh as that weighted blend. The blend changes per output word, so rouge can look mostly at red while voiture looks mostly at car.
Worked example: which source word makes rouge
The decoder is about to emit rouge and scores the three source states as:
Exponentiate each score:
Add them:
Divide to get the attention weights:
The weights sum to about 0.99 after rounding, which is one up to rounding error. The output context is , so roughly two-thirds of the blend is the word red. That is the diagram headline: three source boxes on the left, one blend on the right, and the thickest arrow landing on red.
How we know it worked
There is rarely one right translation, so counting exact matches fails. The BLEU score counts overlapping word groups against a human reference. Take candidate the cat red is sitting against reference the red cat is sitting. Unigrams match 5 of 5, so unigram precision is . Bigrams tell the truth. Candidate bigrams are the cat, cat red, red is and is sitting. Reference bigrams are the red, red cat, cat is and is sitting. Only is sitting overlaps, so bigram precision is . The low bigram score catches the flipped order that unigrams miss. Decoding itself picks each next word with a search over top options, covered in beam search.
Code
The point is the check: rebuild the rouge weights and the bigram precision from the worked example.
import math
scores: list[float] = [2.1, 1.0, 0.2]exps: list[float] = [math.exp(s) for s in scores]total: float = sum(exps)weights: list[float] = [e / total for e in exps]
print([round(e, 2) for e in exps]) # -> [8.17, 2.72, 1.22]print(round(total, 2)) # -> 12.11print([round(w, 2) for w in weights]) # -> [0.67, 0.22, 0.1]
candidate_bigrams: set[str] = {"the cat", "cat red", "red is", "is sitting"}reference_bigrams: set[str] = {"the red", "red cat", "cat is", "is sitting"}precision: float = len(candidate_bigrams & reference_bigrams) / len(candidate_bigrams)print(round(precision, 2)) # -> 0.25The printout matches the hand computation: exps 8.17, 2.72 and 1.22 sum to 12.11, weights 0.67, 0.22 and 0.10, and bigram precision 0.25. Change the first score from 2.1 to 0.2 and all three weights flatten toward one-third each.
Watch Out For
Stuffing long sentences through one fixed vector
Early reader-writer models without attention compress fifty words into one fixed list of numbers. The tail survives and the head fades, so the translation drops the subject and keeps the ending.
Symptom: quality falls steeply past twenty words while short sentences stay fine. Fix: keep per-word encoder states and add the attention blend from attention mechanism instead of one static vector, or move to transformers where every step can reread the source.
Reading BLEU as fluency
A candidate can score high on unigram overlap while scrambling order, as our 1.00 unigram against 0.25 bigram example shows. Teams that track only one number ship word salad that looks good on a dashboard.
Symptom: BLEU climbs while human readers complain about grammar. Fix: report bigram and trigram overlap alongside unigrams, and spot-check outputs by reading them, not just averaging them.
The Quick Version
- Machine translation carries sentence meaning across languages instead of swapping words one by one.
- The encoder reads the, red and car into per-word states while the decoder writes voiture rouge one word at a time.
- Scores 2.1, 1.0 and 0.2 exponentiate to 8.17, 2.72 and 1.22, sum to 12.11, and give weights about 0.67, 0.22 and 0.10.
- The rouge step blends mostly the red state, which is why the adjective lands after the noun in French.
- Candidate the cat red is sitting scores 1.00 on unigrams but 0.25 on bigrams against the red cat is sitting, so bigrams catch flipped order.
- One fixed context vector forgets long inputs, so production systems let every output step reread the source with attention.