Skip to content
AI360XpertBeta

Instruction Tuning

Instruction tuning turns a base model from a text continuer into a request answerer. You show it prompt-plus-ideal-answer pairs, and it learns that the job is to answer, not to extend.

Instruction tuning computes loss on response tokens only, so the model learns to answer instead of continue.
Instruction tuning computes loss on response tokens only, so the model learns to answer instead of continue.

Why Does This Exist?

A base model is a continuer. Hand it Summarize this email: and it may reply with more things shaped like your prompt: another instruction, a second email, the start of a list. Nothing is broken. Predicting likely continuations is exactly what pretraining paid it to do, and the background on that training is in how LLMs work.

But you didn't ask for a continuation. You asked for the summary. Instruction tuning is the short supervised phase that repoints the model: same knowledge, new job. After it, the most likely tokens following your request are the answer, not more requests.

This page covers that repointing: the pair format, the masked loss, and one loss value worked by hand. Building the dataset itself belongs to fine-tuning data preparation. The formatting wrapper around each pair belongs to chat templates.

Think of It Like This

The new barista who memorized the recipe book

She has read every recipe, so she knows every drink on the menu. But on her first shift a customer says "oat latte, please," and she replies "cappuccino, please. Mocha, please." She is continuing the stream of orders she overheard, not filling yours.

So you retrain her for one afternoon. A colleague plays customer, she answers, and you correct only her answers, never the orders. Within a day she gets it: when someone talks to her, the job is the drink, not the echo.

Where it stops: the barista practices a physical skill and her hands get faster. Your model practices nothing but predicting text. Tuning gives it no new facts and no new abilities. It only changes which continuation wins.

How It Actually Works

The setup: pairs, templates, masks

Each training item is a pair. Call the prompt tokens xx and the ideal answer tokens yy. The pair is wrapped in a chat template, the text format that marks who said what, then fed through the model left to right exactly as in pretraining.

The one change is which tokens count. Every prompt token gets loss weight 00. Every answer token gets weight 11. The model still reads the prompt. It just never pays for mispredicting it.

The loss: cross-entropy on answers only

Write TT for the number of answer tokens, yty_t for the tt-th answer token, and ctc_t for everything visible at that position: the prompt plus the answer tokens so far. Write pθp_\theta for the model's predicted probability, where θ\theta means the weights. The tuning loss is:

LSFT=−1T∑t=1Tlog⁡pθ(yt∣ct)L_{\mathrm{SFT}} = -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta(y_t \mid c_t)

In plain words: for each answer token, take the log (natural log) of the probability the model gave the correct token, negate it, and average over the answer tokens. Prompt tokens never enter the sum, so no update ever tells the model to get better at predicting your question.

Because the base model already knows grammar, facts, and style from pretraining, this phase is short: a comparatively small curated set, a small step size, a few passes. You aren't pouring knowledge in. You're rewarding one behavior until it becomes the default.

That sets the trade-off. Instruction tuning buys obedience cheaply and sells nothing it doesn't own: anything the base model never learned can't be tuned out of it. If your model lacks the facts, reach for retrieval or more pretraining, not more SFT. And when answers must be ranked by taste rather than imitated from examples, that's the next phase, preference tuning, covered in DPO and RLHF.

Worked example

Say the answer has T=3T = 3 tokens and the model assigns the correct tokens probabilities 0.700.70, 0.500.50, and 0.800.80.

Negate each natural log:

−ln⁡(0.70)=0.357,−ln⁡(0.50)=0.693,−ln⁡(0.80)=0.223-\ln(0.70) = 0.357, \quad -\ln(0.50) = 0.693, \quad -\ln(0.80) = 0.223

Average them:

LSFT=0.357+0.693+0.2233=1.2733=0.424L_{\mathrm{SFT}} = \frac{0.357 + 0.693 + 0.223}{3} = \frac{1.273}{3} = 0.424

So the loss is 0.4240.424, the number drawn in the diagram above. Two things to notice. The middle token carries most of it (0.6930.693 of 1.2731.273), so the next update pushes its probability up hardest. And the prompt contributes exactly 0.0000.000 however badly the model would have predicted it. If the model had put 0.990.99 on all three answer tokens, the loss would be −ln⁡(0.99)≈0.010-\ln(0.99) \approx 0.010. Confidence in the right answers is the whole game.

Code

The point is the mask: six tokens go in, three count. This replays the worked example as masked mean negative log likelihood over answer tokens only.

import math
token_nll: list[float] = [1.204, 0.511, 0.357, 0.357, 0.693, 0.223]is_answer: list[bool] = [False, False, False, True, True, True]
counted: list[float] = [n for n, a in zip(token_nll, is_answer) if a]loss: float = sum(counted) / len(counted)print([round(v, 3) for v in counted])  # -> [0.357, 0.693, 0.223]print(round(loss, 3))                  # -> 0.424

The first three entries are prompt surprise the model felt but never pays for. The last three are the worked example's values, and their mean is 0.4240.424, matching the hand computation above.

Watch Out For

You trained on the prompts too

Symptom: the tuned model parrots instruction phrasing, completes your question with more questions, or quotes training prompts back at you. It learned the prompts as text to reproduce, because every token paid loss.

The fix is the mask. Confirm prompt tokens carry weight zero in your data collator, then eval on held-out prompts before and after the fix. If prompt-region loss is anything but excluded, the run taught the wrong lesson.

Agreeable data makes an agreeable liar

Symptom: the model nods along with false premises in your questions, then builds on them confidently. If every demonstration answer is cheerful agreement, the tuned behavior is agreement, not honesty.

Mix in corrections, refusals, and answers that contradict the prompt. Then probe with leading questions on purpose: ask it to confirm something false and check that it pushes back.

The Quick Version

  • Instruction tuning is supervised fine-tuning on prompt plus ideal-answer pairs. It turns a continuer into an answerer.
  • Loss is computed on answer tokens only. Every prompt token carries weight zero.
  • The objective is mean negative log likelihood over the TT answer tokens: LSFT=−(1/T)∑log⁡pθL_{\mathrm{SFT}} = -(1/T) \sum \log p_\theta.
  • An answer at token probabilities 0.70, 0.50 and 0.80 gives loss 0.424, and the weakest token dominates the update.
  • Tuning teaches obedience, not facts. It can't surface knowledge pretraining never stored.
  • Preference ranking comes after imitation. When examples stop helping, the next step is preference tuning.