Chain-of-Thought Training
You train the model on worked solutions instead of bare answers, so every intermediate step gets graded. The model learns to show its work because guessing the final answer alone no longer earns full credit.
Why Does This Exist?
Asking a model to think step by step at prompt time helps, but it is a request, not a habit. Take the hint away and most models snap back to blurting out an answer. If you want stepwise work to be the default, you have to put it in the weights.
The obvious training setup works against you here. When every example is question in, final answer out, the only thing that earns loss is the answer token or two. A lucky guess gets full credit and a careful derivation gets none, so the model learns that guessing pays. Multi-step problems punish that bargain first, because one wrong turn early ruins the whole answer with no partial credit along the way.
This page covers the training fix: fine-tuning on reasoning traces, full worked solutions where the steps themselves are graded. The prompt-time version of the trick lives in chain-of-thought prompting. What this training produces at serving time is a reasoning model. The formatting and masking plumbing underneath is fine-tuning data preparation.
Think of It Like This
Wiring a fuse box with the checklist, not just the lit hallway
You are training an apprentice electrician. The exam question is a dark hallway. The final answer is the hallway lit. Standard training shows the apprentice the dark hallway, then the lit hallway, and grades only whether the lights came on. Any wiring that happens to work, safe or not, gets full marks.
Chain-of-thought training hands over the senior electrician's checklist instead: kill the breaker, test the dead wire, land each conductor, torque each lug, label each circuit, and only then flip the breaker. The inspector ticks off every line, so a lucky guess with sloppy wiring cannot pass and careful work earns credit at each step. After enough graded checklists, the apprentice reaches for the checklist unprompted.
Where it stops: a paper checklist cannot lie, but a model trace can. A neat five-step derivation with one invented step reads as confident as a correct one, and the inspector only catches it if someone checks the steps against reality. That is why traces get filtered and tested, never trusted on style.
How It Actually Works
A training example is a triple. Call the question tokens , the reasoning-step tokens , and the final-answer tokens . Here is the count of trace tokens and is the count of answer tokens. The model has parameters , which is just the full set of weights, and it defines a next-token probability , meaning the chance it assigns to token given everything before it. Call the graded history before position by the name .
What the training example looks like
Laid out, one example reads: question, then trace, then answer. The question is the setup and it is masked, which means it contributes no loss; you can see why masking matters in fine-tuning data preparation. The trace is the worked solution in order, and is the short final answer. A tiny example: is "5 + 10 = ?", is "start with 5, add the 10, combine to 15", and is "15".
The loss that teaches the steps
Supervised fine-tuning minimizes cross-entropy, which for one example is the negative log-likelihood (NLL) over the graded tokens:
In plain words: walk through every trace token and every answer token, take the log of the probability the model gave the correct token, negate it, and add it up. The symbol means all graded tokens before position . Low probability on a step means large loss on that step, so the gradient pushes the weights toward producing that step next time. The prompt never appears in the sum, which is exactly what masking enforces.
Worked example with four tokens
Say the trace has three tokens and the answer has one, and the model assigns them these correct-token probabilities: , , , and for the answer. Convert each with , using the natural log:
- for the final answer token
Add them: . That total is the example loss. Divide by the graded tokens and the mean loss is , which corresponds to a perplexity of about (perplexity is raised to the mean loss, so lower means more confident). Now compare: training on the answer alone would grade a single token with loss . The trace version grades four tokens with loss , and most of that signal sits on the steps. That is the whole trick: steps that used to be invisible to the loss now move the weights.
Two honest costs come with it. Each example now carries several times more graded tokens, so you pay more compute and memory per example than answer-only tuning, as in full fine-tuning. And the method assumes the traces are correct. A trace with a wrong middle step teaches the wrong procedure with full confidence, which is why teams filter traces by checking the steps, not just the final answer, and often follow this stage with outcome-based reinforcement learning such as GRPO. Skip this training when the task has no intermediate logic to teach: single-fact recall is cheaper and better served by bare answers.
Code
This snippet recomputes the worked example, so you can check the arithmetic instead of trusting it. It uses only the standard library.
import math
def nll(probs: list[float]) -> float: """Negative log-likelihood summed over the graded tokens.""" return -sum(math.log(p) for p in probs)
def perplexity(probs: list[float]) -> float: """exp of the mean NLL over the graded tokens.""" return math.exp(nll(probs) / len(probs))
trace_probs = [0.70, 0.50, 0.80, 0.40]print(round(nll(trace_probs), 4)) # -> 2.1893print(round(perplexity(trace_probs), 4)) # -> 1.7286print(round(-math.log(0.40), 4)) # -> 0.9163The three outputs agree with the hand computation above: the full trace plus answer costs , the perplexity is about , and the answer-only alternative costs just on one token.
Watch Out For
Neat steps that do not entail the answer
The symptom: the model writes a tidy five-step derivation, every step reads well, and the conclusion does not follow from the steps. You will see this when a trace invents a fact mid-derivation ("so the fee is clearly waived") and sails on. Fluency feels like proof, and reviewers wave it through.
Treat style as guilty until proven innocent. Filter training traces by executing or checking the steps, for instance recomputing the arithmetic and confirming each cited value, and keep only traces whose steps actually produce the answer. At serving time, spot-check derivations on fresh problems the same way. If polished-but-wrong traces keep slipping past, that is the signal to add a reinforcement-learning stage that rewards verified outcomes, not just imitated style.
Letting the loss grade the question tokens
The symptom: training loss falls nicely, yet the model gets worse at following new questions and starts echoing prompt phrasing back at you. It memorized the questions because you paid it to predict them.
Never train on the prompt tokens. Mask out of the loss so the sum runs over trace plus answer only, exactly as the equation above is written. If your framework has a label-masking flag, set prompt labels to the ignore value and confirm with one batch that prompt positions contribute zero before launching the run.
The Quick Version
- Chain-of-thought training fine-tunes on question, trace, and answer triples, so the loss grades every reasoning step instead of only the final answer.
- The loss is negative log-likelihood summed over trace plus answer tokens, with the question tokens masked out of the sum.
- In the worked example, four graded tokens at probabilities 0.70, 0.50, 0.80, and 0.40 give a total loss of 2.1893, against 0.9163 for the answer token alone.
- It costs more graded tokens per example than answer-only tuning, and it teaches whatever procedure the traces contain, correct or not.
- Skip it for single-fact recall with no logic to show; use it where the steps are the skill.