Skip to content
AI360XpertBeta

Pretraining

Train a model to guess each next word across a huge pile of raw text, and the guessing game forces grammar, facts, and style into the weights. The result is a base model: fluent, knowledgeable, and not yet taking orders.

Pretraining turns a raw corpus into a base model by grading every next-token guess, so one page of text supplies thousands of training labels
Pretraining turns a raw corpus into a base model by grading every next-token guess, so one page of text supplies thousands of training labels

Why Does This Exist?

You want an assistant that writes Python, sums up a contract, and chats in French. The straightforward way to build one is supervised learning: gather input-output pairs, grade the model, adjust. But good pairs are slow to make and pricey to buy, while raw text sits everywhere for free. Books, pages, forums, manuals. Billions of words, zero labels.

Pretraining turns that unlabeled pile into labels. Take any document, hide each token in turn, and treat the hidden token as the right answer. A 2,000-token page holds about 1,999 such quizzes, and nobody annotated a thing. Run that game across a huge corpus and the model absorbs grammar, facts, and style along the way, because predicting well requires all three. The result is a base model: a fluent continuer that knows a lot and follows no instructions.

This page covers the run itself: the corpus, the objective, the update, and the base model it leaves behind. The objective in full detail lives in next-token prediction. How you score the result with one number lives in perplexity. What turns the continuer into an assistant lives in supervised fine-tuning.

One idea first: the model never sees words. It sees token IDs, chunks of characters learned by a tokenizer, so "Seine" may be one token while a rare name arrives in pieces. Everything below counts tokens, not words.

Think of It Like This

An apprentice who may watch but never cook

A busy kitchen takes on an apprentice with one rule: you may watch, never touch. Service after service they stand in the corner, guessing the next step before it happens. Salt or sugar. Flip or wait. Plate or reduce.

After a few hundred services they stop guessing blindly. Technique rules out most moves, since you never reduce a salad. Memory rules out most of the rest, since this chef always finishes that sauce with lemon. House habit finishes the job, since Friday's menu is never a surprise twice. Three kinds of knowledge, one corner to stand in.

Hand them an order slip now and watch what happens. Nothing. They know every dish and have never taken an order in their life. That's your base model: trained on everything, instructed in nothing.

Where it stops: the apprentice sees, hears, smells, and gets corrected by a chef who wants them to improve. Your model gets text only, and its only correction is the word that happened to come next. Same training shape, far thinner signal.

How It Actually Works

For a document of tokens x1,x2,…,xnx_1, x_2, \ldots, x_n, pretraining asks one question at every position: given everything so far, what comes next? Write the model's answer pθ(xt+1∣x1,…,xt)p_\theta(x_{t+1} \mid x_1, \ldots, x_t): the probability it assigns to each possible next token, given the tokens before it. Here θ\theta (theta) means the model's weights, the numbers training adjusts, and the bar reads "given." Position tt may only look backward, never at its own answer.

The corpus grades itself

Because the text already exists, the label is free: the correct answer at position tt is simply xt+1x_{t+1}, whatever token actually came next. A document of nn tokens yields n−1n - 1 graded guesses. Take seven tokens:

Paris / is / a / city / on / the / Seine

That strip holds six quizzes: "Paris" predicts "is", "Paris is" predicts "a", all the way down to the six-word context predicting "Seine". Scale that strip to a corpus of billions of strips and you've got the densest free supervision in the field. Nothing was labeled by hand. The text was the label all along.

One position, graded

Raw model outputs are scores, one per vocabulary item, called logits and written z1,…,zVz_1, \ldots, z_V, where VV is the vocabulary size. Scores can be negative and sum to nothing in particular, so they pass through softmax first:

pi=ezi∑jezjp_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

That turns any score list into shares that sum to 1. The position's loss is then −log⁡ptrue-\log p_{\text{true}}, the negative log of the true token's share. It's exactly cross-entropy against a single correct token. Confident and right pays almost nothing. Confident and wrong pays a lot.

Worked example

Context: "Paris is a city on the". True next token: "Seine". Suppose the model scores four candidates:

z=[2.0, 1.0, 0.0, −1.0]for [Seine, River, City, and]z = [2.0,\ 1.0,\ 0.0,\ -1.0] \quad \text{for } [\text{Seine},\ \text{River},\ \text{City},\ \text{and}]

Subtract the max (2.0) to keep the exponentials tame, giving [0,−1,−2,−3][0, -1, -2, -3]. Exponentiate: [1.000,0.368,0.135,0.050][1.000, 0.368, 0.135, 0.050], which sums to 1.5531.553. Divide each by the sum:

p=[0.644, 0.237, 0.087, 0.032],sum 1.000p = [0.644,\ 0.237,\ 0.087,\ 0.032], \qquad \text{sum } 1.000

"Seine" takes almost two-thirds of the mass while plausible rivals split the rest. The loss at this position:

−log⁡(0.644)≈0.44-\log(0.644) \approx 0.44

Had the model handed "Seine" only 0.03, the loss would sit near 3.51, which is the mechanism shouting. These are the numbers drawn in the diagram above and printed by the snippet below.

From one guess to the whole run

The run's loss is that position loss summed across every position of every document:

L(θ)=−∑t=1n−1log⁡pθ(xt+1∣x1,…,xt)\mathcal{L}(\theta) = -\sum_{t=1}^{n-1} \log p_\theta(x_{t+1} \mid x_1, \ldots, x_t)

In plain words: grade each guess, add them up, nudge the weights against the gradient, repeat across the corpus. Each nudge is small. There are a staggering number of them, and that repetition is the point: no single update teaches French, but billions of them leave French in the weights.

Three things decide what comes out, and none of them is the equation. The mix of the corpus decides what gets predicted most, which is why data mixing is its own discipline. The split of the budget between model size and token count decides the price twice, once in training and daily in serving, covered in compute-optimal training. And the objective's own assumption decides the failure: it rewards the likely continuation, and most training sentences are true, so likely and true usually coincide. When they part ways, nothing in the loss notices. That gap is structural, with its own page in hallucination mechanisms.

So the trade is plain. You get the densest free supervision available, in exchange for a model with no built-in notion of truth and a bill few labs can pay. Don't pretrain from scratch when you can avoid it: start from an open base model, add facts with retrieval, and teach manners with tuning instead.

Code

Same fixture as above, plus the pairing trick that turns one strip of text into six labels. Plain Python, standard library only.

import math

def pairs(tokens: list[str]) -> list[tuple[list[str], str]]:    return [(tokens[:i], tokens[i]) for i in range(1, len(tokens))]

def softmax(logits: list[float]) -> list[float]:    peak = max(logits)    exps = [math.exp(z - peak) for z in logits]  # shift first, same result, no overflow    total = sum(exps)    return [e / total for e in exps]

tokens = ["Paris", "is", "a", "city", "on", "the", "Seine"]print(len(pairs(tokens)))  # -> 6print(pairs(tokens)[-1][1])  # -> Seine
logits = [2.0, 1.0, 0.0, -1.0]  # Seine, River, City, andprobs = softmax(logits)print([round(p, 3) for p in probs])  # -> [0.644, 0.237, 0.087, 0.032]print(round(sum(probs), 6))  # -> 1.0print(round(-math.log(probs[0]), 2))  # -> 0.44

One strip, six labels, and the hand computation checks out: shares that sum to 1 and a loss of 0.44 on the true token.

Watch Out For

Asking the continuer for favors

A raw base model is an autocompleter, not an assistant. Prompt it with "What is the capital of France?" and it may reply with "What is the capital of Germany?", because it thinks it is looking at a geography quiz, not a question. Nothing is broken. Continuing plausibly is exactly what pretraining paid it to do.

The fix is a different phase, not a better prompt. Teach the shape of answers with supervised fine-tuning, or reach for an instruction-tuned model that someone already taught. Prompt tricks on a base model are paint on an engine that was never installed.

Grading on text the model already read

Benchmarks live on the open internet, and pretraining eats the open internet. If your test questions slipped into the corpus, the model answers from memory and the score lies. The symptom is specific: brilliant benchmarks, ordinary behavior on anything unpublished.

The fix happens before training, not after. Deduplicate against your eval sets, keep a canary string in each benchmark to grep the corpus for, and trust only held-out tests the model provably never saw. The full treatment is in benchmark contamination.

The Quick Version

  • Pretraining turns raw text into its own labels: at each position, the label is whatever token actually comes next.
  • A document of nn tokens yields n−1n - 1 graded guesses, so a 2,000-token page supplies about 1,999 labels for free.
  • Each guess is graded by cross-entropy: scores [2.0, 1.0, 0.0, -1.0] become shares [0.644, 0.237, 0.087, 0.032], and the true token's loss is 0.44.
  • One objective builds grammar, facts, and style together, because predicting well across varied text requires all three.
  • The loss rewards likely continuations, not checked facts, so fluency is no certificate of truth.
  • The run leaves a base model: a fluent continuer that still needs tuning before it follows instructions.