Skip to content
AI360XpertBeta

GPT & Autoregressive Models

GPT reads strictly left to right and trains on exactly one job: guess the next word, over and over, which turns out to teach it far more than that sounds.

Three context tokens score blue at 2.0, green at 1.0 and grey at 0.0, giving shares 0.665, 0.245 and 0.090, so blue is appended and fed back while the causal mask keeps every position from seeing ahead
Three context tokens score blue at 2.0, green at 1.0 and grey at 0.0, giving shares 0.665, 0.245 and 0.090, so blue is appended and fed back while the causal mask keeps every position from seeing ahead

Why Does This Exist?

Building a language model used to mean one labeled setup per task: translation, summarization, question answering, each with its own small dataset. Labeled data is expensive and small, while raw text is nearly unlimited. GPT bets that one unsupervised job run everywhere, predict the next token, teaches grammar, facts, and reasoning as side effects because getting that job right at scale requires all three.

That job needs an architecture that cannot cheat. GPT stacks decoder-only transformer blocks with causal masking, so position tt reads tokens 11 through tt and never t+1t+1. Without the mask the model would copy the answer it already saw. With it, every position in a 1,000-token page supplies one fair training signal in the same forward pass.

This page covers the left-to-right model and its loop. BERT covers the opposite choice, an encoder with no mask that fills hidden words from both sides. Encoder-decoder architectures lines up all three layouts. How LLMs work covers the same training signal at system scale, and next-token prediction covers the sampling policy that turns scores into text.

Think of It Like This

A karaoke screen that only scrolls forward

You sing karaoke where the screen shows only the lines you already sang plus the line you are on. Future lines stay blank until you sing the current one, then the screen appends your line and reveals the next blank. You guess each word from the words behind you, never from the words ahead.

Each sung line is a generated token, the hidden future lines are the causal mask, your guess at the next word is the probability distribution, and appending what you sang is the autoregressive feedback loop.

Where it stops: you hear melody, breathing, and the crowd, and you can ask to replay a line. The model hears none of that and gets no replay. It holds only token statistics from training text, so a fluent guess can still state a false fact with full confidence.

How It Actually Works

The objective in one line

Causal language modeling predicts token t+1t+1 from tokens 11 through tt, at every tt at once. Call the context x1:tx_{1:t}, meaning the tokens seen so far. Call zz the raw scores over the vocabulary at this step, and pp the shares after softmax. Training pays cross-entropy on the true next token. Inference runs the same forward pass one step at a time: predict, append, feed back, repeat.

Worked example: scores 2.0, 1.0, 0.0 become 0.665, 0.245, 0.090

Say the context is the sky is and the model scores three candidates: blue at 2.02.0, green at 1.01.0, and grey at 0.00.0. Softmax exponentiates and divides by the total:

pk=exp⁡(zk)∑jexp⁡(zj)p_k = \frac{\exp(z_k)}{\sum_j \exp(z_j)}

Here pkp_k is the share for candidate kk and zkz_k its raw score. Exponentiate: exp⁡(2.0)≈7.389\exp(2.0) \approx 7.389, exp⁡(1.0)≈2.718\exp(1.0) \approx 2.718, exp⁡(0.0)=1.000\exp(0.0) = 1.000. Add: 7.389+2.718+1.000=11.1077.389 + 2.718 + 1.000 = 11.107. Divide:

pblue=7.389/11.107≈0.665,pgreen≈0.245,pgrey≈0.090p_{blue} = 7.389 / 11.107 \approx 0.665, \quad p_{green} \approx 0.245, \quad p_{grey} \approx 0.090

Blue wins with two-thirds of the weight, not all of it, which is why generation blends rather than always repeating the top word. The diagram draws these same three bars and the feedback arrow that appends blue. Loss on this step if blue is correct is −log⁡(0.665)≈0.41-\log(0.665) \approx 0.41.

Why one job generalizes

Guessing blue after the sky is needs a fact about color. Guessing a verb after she picked up the phone needs grammar. Finishing a worked subtraction needs the running total. None were labeled separately. Each is just what next-token accuracy demands across varied text, so scale turns one narrow loss into broad capability.

The one-token horizon

Training rewards only token t+1t+1. It never scores whether that choice sets up a good token t+4t+4. Longer-range coherence arrives as a byproduct of scale and data, not as a direct reward, which is why sampling policy and later training stages matter for planning and factuality.

Code

The point is the check: run the same three scores through softmax and confirm 0.665, 0.245, and 0.090, then append the winner the way the loop does.

import math

def softmax(logits: list[float]) -> list[float]:    exps: list[float] = [math.exp(z) for z in logits]    total: float = sum(exps)    return [e / total for e in exps]

logits: list[float] = [2.0, 1.0, 0.0]probs: list[float] = softmax(logits)print(f"{probs[0]:.3f} {probs[1]:.3f} {probs[2]:.3f}")  # -> 0.665 0.245 0.090print(f"{-math.log(probs[0]):.2f}")  # -> 0.41
context: list[str] = ["the", "sky", "is"]context.append(["blue", "green", "grey"][probs.index(max(probs))])print(context)  # -> ['the', 'sky', 'is', 'blue']

The printout matches the hand computation: exps 7.389, 2.718, and 1.000 sum to 11.107, shares 0.665, 0.245, and 0.090, loss 0.41, and the top token appended for the next step.

Watch Out For

Reading causal as narrow context

Symptom: a reader assumes a decoder-only model sees little context because it cannot look ahead, and designs prompts as if only the last few words matter. Fixes target the wrong end, such as shortening inputs that were already fine.

Fix: causal blocks only the future. At token 2,000 the model reads tokens 1 through 1,999 in full. Debug weak answers by checking what came before, especially order and recency, not by assuming the window is small.

Mistaking fluency for planning

Symptom: fluent multi-step text is read as deliberate long-horizon planning, so a team ships chain-of-thought answers with no checks and meets confident detours halfway through.

Fix: the loss paid for one token ahead only. Treat longer coherence as useful but unguaranteed, and add the sampling controls in next-token prediction plus outside verification for facts, arithmetic, and multi-step plans.

The Quick Version

  • GPT is a decoder-only, causally masked transformer trained on one job: predict token t+1t+1 from tokens 11 through tt.
  • The same forward pass trains on all positions at once and generates one token at a time by appending each output to its own input.
  • Scores 2.0, 1.0, and 0.0 give shares 0.665, 0.245, and 0.090, so the top token takes two-thirds of the weight rather than all of it.
  • Broad grammar, facts, and reasoning arrive as byproducts of doing that one narrow job well at scale.
  • The objective rewards only the very next token, so longer-range coherence is a byproduct rather than a direct guarantee.