GPT & Autoregressive Models
GPT reads strictly left to right and trains on exactly one job: guess the next word, over and over, which turns out to teach it far more than that sounds.
Why Does This Exist?
Building a language model used to mean one labeled setup per task: translation, summarization, question answering, each with its own small dataset. Labeled data is expensive and small, while raw text is nearly unlimited. GPT bets that one unsupervised job run everywhere, predict the next token, teaches grammar, facts, and reasoning as side effects because getting that job right at scale requires all three.
That job needs an architecture that cannot cheat. GPT stacks decoder-only transformer blocks with causal masking, so position reads tokens through and never . Without the mask the model would copy the answer it already saw. With it, every position in a 1,000-token page supplies one fair training signal in the same forward pass.
This page covers the left-to-right model and its loop. BERT covers the opposite choice, an encoder with no mask that fills hidden words from both sides. Encoder-decoder architectures lines up all three layouts. How LLMs work covers the same training signal at system scale, and next-token prediction covers the sampling policy that turns scores into text.
Think of It Like This
A karaoke screen that only scrolls forward
You sing karaoke where the screen shows only the lines you already sang plus the line you are on. Future lines stay blank until you sing the current one, then the screen appends your line and reveals the next blank. You guess each word from the words behind you, never from the words ahead.
Each sung line is a generated token, the hidden future lines are the causal mask, your guess at the next word is the probability distribution, and appending what you sang is the autoregressive feedback loop.
Where it stops: you hear melody, breathing, and the crowd, and you can ask to replay a line. The model hears none of that and gets no replay. It holds only token statistics from training text, so a fluent guess can still state a false fact with full confidence.
How It Actually Works
The objective in one line
Causal language modeling predicts token from tokens through , at every at once. Call the context , meaning the tokens seen so far. Call the raw scores over the vocabulary at this step, and the shares after softmax. Training pays cross-entropy on the true next token. Inference runs the same forward pass one step at a time: predict, append, feed back, repeat.
Worked example: scores 2.0, 1.0, 0.0 become 0.665, 0.245, 0.090
Say the context is the sky is and the model scores three candidates: blue at , green at , and grey at . Softmax exponentiates and divides by the total:
Here is the share for candidate and its raw score. Exponentiate: , , . Add: . Divide:
Blue wins with two-thirds of the weight, not all of it, which is why generation blends rather than always repeating the top word. The diagram draws these same three bars and the feedback arrow that appends blue. Loss on this step if blue is correct is .
Why one job generalizes
Guessing blue after the sky is needs a fact about color. Guessing a verb after she picked up the phone needs grammar. Finishing a worked subtraction needs the running total. None were labeled separately. Each is just what next-token accuracy demands across varied text, so scale turns one narrow loss into broad capability.
The one-token horizon
Training rewards only token . It never scores whether that choice sets up a good token . Longer-range coherence arrives as a byproduct of scale and data, not as a direct reward, which is why sampling policy and later training stages matter for planning and factuality.
Code
The point is the check: run the same three scores through softmax and confirm 0.665, 0.245, and 0.090, then append the winner the way the loop does.
import math
def softmax(logits: list[float]) -> list[float]: exps: list[float] = [math.exp(z) for z in logits] total: float = sum(exps) return [e / total for e in exps]
logits: list[float] = [2.0, 1.0, 0.0]probs: list[float] = softmax(logits)print(f"{probs[0]:.3f} {probs[1]:.3f} {probs[2]:.3f}") # -> 0.665 0.245 0.090print(f"{-math.log(probs[0]):.2f}") # -> 0.41
context: list[str] = ["the", "sky", "is"]context.append(["blue", "green", "grey"][probs.index(max(probs))])print(context) # -> ['the', 'sky', 'is', 'blue']The printout matches the hand computation: exps 7.389, 2.718, and 1.000 sum to 11.107, shares 0.665, 0.245, and 0.090, loss 0.41, and the top token appended for the next step.
Watch Out For
Reading causal as narrow context
Symptom: a reader assumes a decoder-only model sees little context because it cannot look ahead, and designs prompts as if only the last few words matter. Fixes target the wrong end, such as shortening inputs that were already fine.
Fix: causal blocks only the future. At token 2,000 the model reads tokens 1 through 1,999 in full. Debug weak answers by checking what came before, especially order and recency, not by assuming the window is small.
Mistaking fluency for planning
Symptom: fluent multi-step text is read as deliberate long-horizon planning, so a team ships chain-of-thought answers with no checks and meets confident detours halfway through.
Fix: the loss paid for one token ahead only. Treat longer coherence as useful but unguaranteed, and add the sampling controls in next-token prediction plus outside verification for facts, arithmetic, and multi-step plans.
The Quick Version
- GPT is a decoder-only, causally masked transformer trained on one job: predict token from tokens through .
- The same forward pass trains on all positions at once and generates one token at a time by appending each output to its own input.
- Scores 2.0, 1.0, and 0.0 give shares 0.665, 0.245, and 0.090, so the top token takes two-thirds of the weight rather than all of it.
- Broad grammar, facts, and reasoning arrive as byproducts of doing that one narrow job well at scale.
- The objective rewards only the very next token, so longer-range coherence is a byproduct rather than a direct guarantee.