Supervised Fine-Tuning

SFT keeps training a pretrained model on curated example answers, scoring only the answers, so a fixed response shape becomes its habit instead of a suggestion.

Why Does This Exist?

A pretrained model is good at one thing: continuing text in a way that looks plausible. That is genuinely useful, and it is not the same as doing what you asked. Say you are building a support chatbot with an exact brief: acknowledge the problem in one line, list the next steps as a numbered list, then close with one reassuring line. A base model might nail that shape once, ramble the next time, and drop the numbered list entirely on the third try. Nothing is broken. Nobody ever taught it that this one shape is the shape that counts.

Prompting helps, but it does not lock anything in. Across thousands of differently worded customer messages, a prompt-only setup drifts: the acknowledgment disappears, the list melts into a paragraph, the closing line vanishes. That drift is not a bug in your prompt. It is what happens when the behavior lives in an instruction the model reads instead of in the weights themselves.

Supervised fine-tuning moves the behavior into the weights. You show the model real input-output pairs and keep training it until producing that pattern is just what it does. This page covers the mechanism: the demonstration pairs, the masked loss, and one loss value worked by hand. The decision of whether to fine-tune at all belongs to when to fine-tune. The instruction-following application of this technique has its own page at instruction tuning. The preference step that usually comes after sits at RLHF. The one prerequisite is next-token prediction: the model reads left to right and predicts each next token, and SFT keeps that exact objective while changing which tokens count.

Think of It Like This

A folder of worked examples for a new support agent

You onboard a new hire with a style guide: acknowledge the issue, list the steps, close warmly. Some hires nail it from the memo. Most do not. A written rule and consistent behavior under real pressure are different things.

So instead you hand them a folder of two hundred real customer emails, each with the ideal reply attached, every one written exactly the way you want every reply written. They read pair after pair, this question with that answer, and after enough of them the pattern stops being a rule they consciously follow. It becomes just how they write.

That folder is the demonstration set. Absorbing it is what SFT does to a model.

Where it stops: your new hire also learns facts about your products from those emails, asks questions, and notices when a reply contradicts policy. The model does none of that. It only adjusts probabilities toward the demonstrated phrasing. If the folder contains sloppy replies, it learns sloppiness with the same dedication.

How It Actually Works

Demonstration pairs and a familiar objective

Each training item is a pair: a prompt and the exact answer you want for it. Write the prompt tokens as $x = (x_1, \ldots, x_m)$ , where each $x_i$ is one input token and $m$ is how many there are. Write the ideal answer tokens as $y = (y_1, \ldots, y_n)$ , where each $y_j$ is one answer token and $n$ is the answer length. The model, with weights $\theta$ (theta, every number the training can change), reads the prompt followed by the answer left to right, exactly as in pretraining.

The training objective is the same one pretraining used: next-token prediction. At each answer position $j$ , look at $p(y_j \mid x, y_1, \ldots, y_{j-1})$ , which is the probability the model assigns to the correct next answer token given the prompt and the answer so far. The loss over one pair is the mean negative log likelihood over the answer tokens:

\mathcal{L}(\theta) = -\frac{1}{n} \sum_{j=1}^{n} \log p(y_j \mid x, y_1, \ldots, y_{j-1})

In plain words: for each answer token, take the log (natural logarithm) of the probability the model gave the right token, negate it so confident-correct scores near zero, and average over the $n$ answer tokens. Nothing about the machinery changed from pretraining. What changed is where the correct answer comes from: not whatever happened to appear next in a scraped document, but a demonstration someone wrote or approved because it is the answer they want reinforced.

Why the prompt is masked out

The model must read the whole sequence, prompt included. You cannot condition on a prompt you never saw. But the loss counts only the $n$ answer tokens. Every prompt token gets loss weight $0$ and every answer token gets weight $1$ .

This is deliberate. You will never ask the model to generate a customer message, so spending gradient signal on predicting customer messages burns capacity on a skill nobody needs and dilutes the real objective. Masking concentrates every update on the thing you actually want shaped: what the model produces once the message is already given. The diagram above shows the split for one pair: five prompt tokens with weight 0, three answer tokens with weight 1.

Worked example: three answer tokens

Take a pair with $m = 5$ prompt tokens and $n = 3$ answer tokens. Suppose that, given the prompt and the answer so far, the model assigns the three correct answer tokens the probabilities $0.70$ , $0.50$ and $0.80$ . Negate each natural log:

$-\log(0.70) = 0.357$
$-\log(0.50) = 0.693$
$-\log(0.80) = 0.223$

Add them: $0.357 + 0.693 + 0.223 = 1.273$ . That sum is the total loss on this pair, and the mean per scored token is $1.273 / 3 = 0.424$ . Training nudges $\theta$ to push those three probabilities up, which pushes 0.424 down.

Now see what masking buys. If you scored all 8 tokens and the five prompt tokens had probabilities $0.90$ , $0.85$ , $0.92$ , $0.88$ and $0.95$ , their negative logs add $0.530$ , the total becomes $1.803$ , and the mean over 8 tokens drops to $0.225$ . Same model, same pair, nearly half the reported loss, and most of the gradient now teaches prompt prediction. The masked number 0.424 tells you how the answers are doing. The unmasked 0.225 mostly tells you prompts are easy to predict.

Why a small clean set wins

A demonstration set teaches whatever pattern is actually in it, consistent or not. Two hundred transcripts that all follow the acknowledge-list-reassure shape with zero exceptions teach that shape cleanly, because every example reinforces the same thing. Five thousand scraped transcripts, where one in five skips the numbered list, teach the model that skipping the list is an acceptable one-in-five move. From the model's point of view, a frequent pattern in the data is the correct pattern to learn. For this technique, curation beats volume nearly every time.

That gives you the trade-off, the assumption, and the boundary in one package. The trade-off: SFT buys reliable format and style at the price of a curated dataset plus a training run, and it risks narrowing the model toward your demonstrations. The assumption: your pairs cover the situations the model will meet. When a customer message arrives shaped nothing like your training prompts, the learned pattern can misfire, which is why you hold out a slice of pairs and read real generations before shipping. And when not to use it: SFT locks in style, not facts, so fast-changing knowledge belongs in retrieval rather than weights, and ranking one answer above another belongs to a preference method such as RLHF. If a sharper prompt already holds the format, you have not earned the training run yet.

Code

A minimal version of the masked loss from the worked example: mean negative log likelihood over the masked positions only, plus the unmasked comparison that shows the dilution.

1import math2
3
4def masked_mean_nll(probs: list[float], mask: list[int]) -> float:5    """Mean negative log likelihood over positions where mask is 1."""6    scored = [p for p, m in zip(probs, mask) if m == 1]7    return -sum(math.log(p) for p in scored) / len(scored)8
9
10probs = [0.90, 0.85, 0.92, 0.88, 0.95, 0.70, 0.50, 0.80]11mask = [0, 0, 0, 0, 0, 1, 1, 1]12print(round(masked_mean_nll(probs, mask), 3))  # -> 0.42413print(round(sum(-math.log(p) for p in probs) / len(probs), 3))  # -> 0.225

The first number is the 0.424 from the worked example: three answer tokens, three terms, divided by three. The second is the diluted 0.225 you get when the five prompt tokens join the average. Same model, same pair, different mask.

Watch Out For

Scoring the prompt alongside the answer

Skipping the mask does not crash. It quietly teaches the model to predict input tokens it will never generate, and it flatters your metrics while doing it: in the worked example the reported loss nearly halves, from 0.424 to 0.225, without the answers improving at all. The symptom is slow format learning paired with a loss curve that looks fine, plus generations that echo prompt phrasing. The fix is mechanical: build the mask from the two lengths, assert its sum equals the answer length before every run, and log the masked and unmasked means separately so dilution can never hide inside one number.

Buying volume instead of curation

Scraping every past transcript to reach thousands of pairs feels like progress, and it trains an inconsistent model. If a fifth of the transcripts skip the numbered list, the model learns that skipping it is correct a fifth of the time, and no training setting un-teaches a pattern the data keeps demonstrating. The symptom shows up in evaluation: most replies perfect, a steady minority breaking format in ways that look random but match the data's own exception rate. The fix is to hold the line on consistency: a smaller hand-checked set with zero exceptions, a format check over every pair before training, and a held-out slice whose generations you read with your own eyes.

The Quick Version

Supervised fine-tuning keeps training a pretrained model on demonstration pairs: a prompt plus the exact answer you want for it.
It reuses the pretraining objective, next-token prediction, but computes the loss only over the answer tokens while the prompt stays unread-for-scoring context.
In the worked example, three answer tokens at probabilities 0.70, 0.50 and 0.80 give a masked mean loss of 0.424; scoring the prompt too would dilute it to 0.225.
A small consistent demonstration set teaches a format better than a large noisy one, because the model learns the exceptions too.
SFT locks in style and format, not new facts; fast-changing knowledge belongs in retrieval, and ranking answers belongs to a preference method such as RLHF.