Next-Token Prediction

You train a model on one job, guess the next word from the words before it, and getting that guess right across enough text forces grammar, facts, and habits of style into the same weights.

At each position the model turns the words so far into one probability distribution, and the loss is the negative log of the true next token's share

Why Does This Exist?

Say you want a model that writes fluent English. The obvious route is supervised learning: collect input-output pairs, grade the model, adjust. But good pairs are slow and pricey to make, while raw text sits everywhere: books, pages, forums, manuals. Billions of words, zero labels.

Next-token prediction turns that unlabeled pile into labels for free. Take any sentence, hide each word in turn, and treat the hidden word as the right answer. A 2,000-token page holds about 1,999 such quizzes, every one graded against text that already exists. Nobody annotated anything.

This page covers the training objective itself: the distribution, the loss, and one worked example. How you draw tokens from the model at inference lives in decoding strategies. How you score a trained model with a single number lives in perplexity.

One idea first: the model never sees words. It sees token IDs, chunks of characters learned by a tokenizer, so "Seine" may be one token while a rare word arrives in pieces. If that split is new to you, read byte-pair encoding first, then come back. Everything below counts tokens, not words.

Think of It Like This

Reading with your thumb over the next word

Grab a printed page and cover the next word with your thumb. Guess it, slide your thumb, check. Do that down the whole page.

After a few lines you stop guessing blindly. Grammar rules out most words, since a noun has to sit here. Facts rule out most of the rest, since Paris sits on the Seine and not the Thames. Habit finishes the job, since a travel guide continues the sentence one way and a lease another. Three kinds of knowledge, one thumb game.

Here is where the analogy stops. You walked in already knowing English, and your wrong guesses cost you nothing. The model starts from noise, and every miss nudges its weights. Same game, much higher stakes: played across enough text, the game becomes the training run.

How It Actually Works

For a sequence of tokens $x_1, x_2, \ldots, x_n$ , the model outputs a probability distribution over the whole vocabulary at each position. Write it $p_\theta(x_{t+1} \mid x_1, \ldots, x_t)$ : the chance it assigns to each possible next token, given everything so far. Here $\theta$ (theta) means the model's weights, the numbers training adjusts, and the vertical bar reads "given."

Two rules keep the game honest. First, position $t$ may only look backward. A causal mask blocks every later token, so no prediction peeks at its own answer. Second, training grades every position, not one per document. The loss is cross-entropy summed across positions:

\mathcal{L}(\theta) = -\sum_{t=1}^{n-1} \log p_\theta(x_{t+1} \mid x_1, \ldots, x_t)

In plain words: at each position, take the probability given to the token that actually came next, take its log, negate it, add it up. Confident and right pays almost nothing. Confident and wrong pays a lot.

The loss at one position

Raw model outputs are scores, one per vocabulary item, called logits and written $z_1, \ldots, z_V$ , where $V$ is the vocabulary size. Scores can be negative and sum to nothing in particular, so they pass through softmax first:

p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

That turns any score list into probabilities that sum to 1. The position's loss is then $-\log p_{\text{true}}$ , the negative log of the true token's share. Since the truth is a single token, this is exactly cross-entropy against a one-hot target. One number per position, all of them added into the training signal.

Worked example

Context: "Paris is a city on the". True next token: "Seine". Suppose the model scores four candidates:

z = [2.0,\ 1.0,\ 0.0,\ -1.0] \quad \text{for } [\text{Seine},\ \text{River},\ \text{City},\ \text{and}]

Subtract the max (2.0) to keep the exponentials tame, giving $[0, -1, -2, -3]$ . Exponentiate: $[1.000, 0.368, 0.135, 0.050]$ , which sums to $1.553$ . Divide each by the sum:

p = [0.644,\ 0.237,\ 0.087,\ 0.032], \qquad \text{sum } 1.000

The model hands "Seine" almost two-thirds of the mass while spreading the rest over plausible rivals. The loss at this position:

-\log(0.644) \approx 0.44

Had the model given "Seine" only 0.03, the loss would be about 3.51, which is the mechanism's way of shouting. These are the same numbers drawn in the diagram above and printed by the snippet below.

Why one loss teaches many skills

Getting "Seine" right takes more than grammar. Plenty of nouns fit after "on the", so grammar alone leaves the model choosing among thousands. The fact picks the river, since Paris sits on the Seine. The register shapes the phrasing around it, since a travel guide continues differently than a lease. Gradient descent cannot tell these apart and does not need to. Each is a necessary condition for predicting well across varied text, so minimizing the one loss builds all three into the weights as a side effect.

What it assumes, and where that breaks

The objective assumes tomorrow's text looks like the training text. It rewards the likely continuation, and most training sentences are true, so likely and true usually coincide. When they part ways, nothing in the loss notices: no term checks facts, only fit. That gap is structural, and it has its own page in hallucination mechanisms.

So skip this objective where it does not fit. If your task needs both directions of context, like filling in a blank mid-sentence, a masked objective suits it better. If you need guaranteed facts, pair the model with retrieval instead of hoping the weights contain them. The trade is plain: the densest free supervision in the field, in exchange for a model with no built-in notion of truth.

Code

The same fixture as above, computed from scratch in plain Python. Nothing beyond the standard library, so you can run it anywhere.

1import math2
3
4def softmax(logits: list[float]) -> list[float]:5    peak = max(logits)6    exps = [math.exp(z - peak) for z in logits]  # shift first, same result, no overflow7    total = sum(exps)8    return [e / total for e in exps]9
10
11logits = [2.0, 1.0, 0.0, -1.0]  # Seine, River, City, and12probs = softmax(logits)13print([round(p, 2) for p in probs])  # -> [0.64, 0.24, 0.09, 0.03]14print(round(sum(probs), 6))          # -> 1.015print(round(-math.log(probs[0]), 2))  # -> 0.44, the loss when Seine is the true next token

The outputs match the hand computation up to rounding: shares that sum to 1, and a loss of 0.44 for the true token's 0.644 share.

Watch Out For

Reading fluency as fact-checking

A smooth sentence feels checked, and this objective never checks anything. It grades fit to training text, so a model can state a wrong year or a made-up citation in perfect prose and collect near-zero loss for it, provided similar sentences appeared in training.

The symptom is confident error on specifics: names, dates, numbers. When the answer matters, ground the model with retrieval or verify the specifics yourself. Fluency is evidence of likely phrasing, nothing more.

Training on perfect prefixes, generating from your own words

During training the model always reads real text before each guess, a setup called teacher forcing. At inference there is no answer key: each generated token becomes context for the next, so one bad pick poisons everything after it. Small errors compound, and long generations drift or loop.

The symptom is quality that decays over a long output while short answers stay sharp. The practical fixes: keep generations short and chain them, sample with temperature or top-p instead of always taking the top token (see decoding strategies), and judge the model on full generations, never on loss alone.

The Quick Version

Next-token prediction turns raw text into its own labels: at each position, the label is whatever token actually comes next.
The model outputs one probability distribution per position, and cross-entropy grades the share given to the true token.
Scores [2.0, 1.0, 0.0, -1.0] become shares [0.644, 0.237, 0.087, 0.032], so the true token's loss is 0.44.
One objective builds grammar, facts, and style together because predicting well across varied text requires all three.
The loss rewards likely continuations, not checked facts, so fluent wording is no certificate of truth.
Training always feeds real prefixes, while generation feeds the model its own words, so errors can compound over long outputs.