Post-Training Evaluation
Training loss can fall while the model gets worse where it counts. You grade the tune on held-out tasks against a prompted baseline, then check general skills for what it forgot.
Why Does This Exist?
Your loss curve looks perfect and your users disagree. That gap is why this page exists.
Say you tune a support model on a few thousand of your own tickets. The loss falls each epoch because the model gets better at repeating your training format. In production it still cites the wrong policy, drops your greeting, or answers worse than the base model did with five good examples in the prompt. Training loss measured memory. It never measured whether the tune beat prompting, or whether general skills survived.
Post-training evaluation is the check that answers both. You score the tuned model on held-out tasks against the base model with its best prompt, then you retest general skills for forgetting. If either side fails, you don't ship.
You need one idea first. Supervised fine-tuning keeps training a pretrained model on your demonstration pairs, with loss on the completion only. Everything here assumes that setup. The forgetting mechanism gets its own page in fine-tuning forgetting. The scoring trick for open answers lives in LLM-as-a-judge. The decision to tune at all belongs to when to fine-tune. This page covers the protocol that judges the tune once it exists.
Think of It Like This
A restaurant test after a specialty course
You run a solid neighborhood kitchen. You send your lead cook to a three week pastry course.
When they return, you run two tastings. First, you serve their new tart next to the tart they baked before the course with your best recipe card in hand. If the new tart isn't clearly better, the course didn't earn its cost. That side by side is the prompted baseline. Second, you ask for a plain omelet during the lunch rush. If the tart shines but the omelet now sticks, scrambles wrong, and slows the line, the specialty came at the price of the everyday menu. That second test is regression.
Where it stops: a cook knows they trained and can tell you what feels rusty. A model can't. Forgetting shows only in scores, so you've got to run both tastings even when the first one thrills you.
How It Actually Works
You freeze two test sets before you judge anything, and you never let either set leak into training. One set covers your task. The other covers general skills you refuse to break. Same sets, same prompts, same grader for both models. Change one of those and the comparison means nothing.
The two sets you need
Call the number of held-out task examples, meaning prompts the model never trained on. Call the count it answers right. Accuracy is the share right:
Here is accuracy, is correct count, and is total count. For the task set, write for the tuned model and for the base model with its best few-shot prompt. The task gain is the difference in points:
Here is task gain, positive when the tune helps. Do the same on general questions with , meaning held-out general examples, and call the difference , meaning general drift. Positive drift means general skills held. Negative drift means forgetting.
Why the baseline must be prompted
The raw base is a straw man. It never saw your format, so almost any tune beats it. The honest rival is the base with your best prompt: clear instructions plus a few worked examples, the setup in few-shot prompting. If your tune can't clearly beat that rival on the same frozen task set, you've bought a training pipeline for what a paragraph already did. Keep the prompt fixed across both models, or you are grading prompts instead of weights.
How the regression check reads forgetting
Tuning drags shared weights toward your task, and general skills can slide with them. That slide is fine-tuning forgetting. So you rerun a small general set: plain questions, short instructions, simple coding or reasoning items your users still hit. Score it exactly as before. A small dip may be acceptable. A large one means the model turned brittle, and it will fail on the first odd question in production even while your task number glows.
Worked example
Take 40 held-out tickets and 30 general questions. Numbers small enough to check by hand, and they match the diagram above.
On tickets, the prompted base gets right out of :
The tuned model gets right:
So , or plus 12.5 points on five extra correct tickets.
On general questions, the base gets of :
The tuned model gets of :
So , or minus 6.7 points on two lost questions. The tune clearly helped the task and clearly cost general skill. Whether you ship depends on how much that general slip hurts your users.
What it costs, when to hold back, and what you assume
The trade is effort for truth. A full gate with frozen sets, a fixed prompt, and human or judge grades takes time to build and rerun, and it slows every release. Skipping it buys speed and pays in silent breakage: a tune that aces training format while forgetting how to follow plain instructions.
Don't run the full human panel on every checkpoint. Use a cheap automatic slice during iteration, then run the full suite at release gates and before any retrain ships. Skip the general suite only when the deployment truly cannot reach old skills, which is rare for chat or support. A narrow classifier with no free text input is the exception, not the rule.
The method assumes both sets look like deployment and stayed unseen. When that fails, through leaked test items or a stale set from last quarter, the numbers lie in your favor. Leaked items read as skill. Stale items miss the questions users ask now. That failure mode has its own name in benchmark contamination.
Code
The point of this snippet is the arithmetic behind the verdict: two accuracies and two deltas from raw counts.
def accuracy(correct: int, total: int) -> float: return correct / total
task_base = accuracy(30, 40)task_tuned = accuracy(35, 40)gen_base = accuracy(24, 30)gen_tuned = accuracy(22, 30)
print(f"{task_base:.3f}") # -> 0.750print(f"{task_tuned:.3f}") # -> 0.875print(f"{task_tuned - task_base:+.3f}") # -> +0.125print(f"{gen_tuned:.3f}") # -> 0.733print(f"{gen_tuned - gen_base:+.3f}") # -> -0.067They're the same 0.750, 0.875, and 0.733 the worked example computed by hand, with a task gain of plus 0.125 and a general drift of minus 0.067.
Watch Out For
You beat the raw base and call it a win
Symptom: the tune shows a big lead over the plain base, then someone pastes five good examples into the base prompt and the lead shrinks to almost nothing. You nearly shipped a pipeline for a paragraph.
Fix: Freeze one strong few-shot prompt and one held-out task set before you train, and grade both models on exactly that. If the tuned lead over the prompted base is thin or wobbly across reruns, keep the prompt.
Test examples leak into training
Symptom: eval accuracy looks near perfect on a slice, yet users hit basic errors on the same kind of question. One duplicated ticket or pasted benchmark item taught the model the answer instead of the skill.
Fix: hash and dedup train against every eval set, keep one eval set private to the release gate, and recheck overlap after each data refresh. A leaked item voids its score.
The Quick Version
- Post-training evaluation grades a tune on two frozen sets: held-out task prompts and general skill prompts.
- The task score must beat the base model with its best few-shot prompt, not the raw base model.
- A tune that moves tickets from 30 of 40 to 35 of 40 gains 12.5 points on the task.
- A tune that moves general questions from 24 of 30 to 22 of 30 drops 6.7 points on general skill.
- A leaked test item measures memory, so dedup by hash and keep one eval set private.