Language Models

A language model learns which words tend to follow which, so it can score every possible next word and generate text one probable word at a time.

Logits 2, 1 and 0 become weights 0.665, 0.245 and 0.090, and the sampler picks the top token to continue the text.

Why Does This Exist?

Early systems tried to write language down as rules: dictionaries, syntax trees, hand-built grammars. Speech kept breaking them. "Recognize speech" and "wreck a nice beach" sound alike, and no rulebook settles which one a caller meant. Rules describe language. They never made a machine fluent in it.

The fix reframed language as probability. A language model scores how likely each word sequence is, so "recognize speech" beats its sound-alike by sheer frequency in real conversation. That single score powers transcription, completion, translation fluency, and every modern assistant. Predict the next word well enough and whole paragraphs follow.

This page covers the scoring itself: the chain rule, the short-window shortcut, the neural upgrade, and the sampler. Raw text becomes pieces in tokenization, pieces become vectors in word embeddings, and the finished scores drive question answering. The one prerequisite is small: a token is a piece of text the model treats as one unit, often a word or part of one. Everything else builds on scoring what comes next.

Think of It Like This

The most observant autocomplete you know

Say "I went to the bank to deposit my" to a well-read friend and they will guess "money" before you finish. Not "river". The word "deposit" settled it. Your friend counted nothing consciously, but years of reading taught them which endings follow which beginnings.

A language model is that friend with a perfect memory of billions of pages. It tracks every "deposit my" it ever saw and what followed. Scale that counting from one friend to the whole web and autocomplete starts writing essays. Same trick, vastly more reading.

Where it stops: your friend knows what money is. The model does not. It knows "money" follows "deposit my" far more often than "river" does. Frequency mimics understanding until facts matter, and that gap is where its confident errors come from.

How It Actually Works

A language model takes a token sequence and returns a probability distribution over the vocabulary for the next token. Three ideas build it.

The chain rule

The probability of "the cat sat" factors left to right. Here $P$ is probability and $\mid$ reads "given":

P(\text{the, cat, sat}) = P(\text{the}) \times P(\text{cat} \mid \text{the}) \times P(\text{sat} \mid \text{the, cat})

Exact histories explode. A paragraph-long condition never repeats in training, so its count is zero and the product dies. Something must shrink the context.

The short-window shortcut

Classical models assume only recent words matter. A bigram ( $N = 2$ ) looks one word back:

P(\text{the, cat, sat}) \approx P(\text{the}) \times P(\text{cat} \mid \text{the}) \times P(\text{sat} \mid \text{cat})

Each term is a count ratio. On the toy corpus "the cat sat", "the cat ran", "the dog sat" (9 tokens, "the" 3 times): $P(\text{the}) = 3/9 \approx 0.333$ , $P(\text{cat} \mid \text{the}) = 2/3 \approx 0.667$ , $P(\text{sat} \mid \text{cat}) = 1/2 = 0.5$ . Multiply:

0.333 \times 0.667 \times 0.5 \approx 0.111

The sentence scores about $0.111$ . Counting works, but unseen pairs still score zero, and "dog" learns nothing from "cat" no matter how similar they are.

The neural upgrade and the sampler

Neural models replace counts with vectors. Tokens become embeddings, layers mix long contexts, and a final layer emits one raw score (a logit) per vocabulary word. Softmax turns logits into probabilities that sum to $1$ . For logits $[2, 1, 0]$ : exponentials $7.39$ , $2.72$ , $1.00$ sum to $11.11$ , giving weights about $0.665$ , $0.245$ , $0.090$ , the bars the diagram draws. Then the sampler decides: greedy takes the top weight, temperature sampling rolls the dice so lower weights sometimes win and text stays varied.

Code

This trains a tiny bigram on three sentences and scores the running example, using only the standard library.

1corpus = ["the cat sat", "the cat ran", "the dog sat"]2tokens: list[str] = [w for s in corpus for w in s.split()]3bigrams: dict[tuple[str, str], int] = {}4for a, b in zip(tokens, tokens[1:]):5    bigrams[(a, b)] = bigrams.get((a, b), 0) + 16def following(word: str) -> dict[str, float]:7    total = sum(c for (a, _), c in bigrams.items() if a == word)8    return {b: c / total for (a, b), c in bigrams.items() if a == word}9print(f"{following('the')['cat']:.3f}")  # -> 0.66710print(f"{following('cat')['sat']:.3f}")  # -> 0.50011print(f"{3 / 9 * 2 / 3 * 1 / 2:.3f}")  # -> 0.111

Watch Out For

One unseen pair zeroing the whole sentence

Bigram probabilities multiply, so a single pair absent from training scores zero and the entire sentence scores zero with it, however natural the rest reads. The symptom is fluent test sentences reporting probability exactly 0. Fix it with smoothing that reserves mass for the unseen (even simple add-one), or back off to shorter contexts instead of trusting raw counts.

Reading likelihood as truth

The model reports what usually follows, not what is true, so frequent misconceptions in training become confident completions. The symptom is fluent, specific, wrong answers with no warning attached. Fix it by grounding generation in retrieved sources and verifying claims outside the model, never by sampling harder.

The Quick Version

A language model scores sequences by predicting each next word from the words before it.
The chain rule factors the joint probability; bigrams approximate it with one-word histories counted from text.
In the worked example, "the cat sat" scores about 0.111 from ratios 0.333, 0.667 and 0.5.
Neural models swap counts for embeddings and long contexts, then softmax turns logits 2, 1, 0 into weights 0.665, 0.245 and 0.090.
The sampler picks from those weights, greedily or with temperature, and likelihood must never be mistaken for truth.