Skip to content
AI360XpertBeta

Fine-Tuning Data Preparation

Your fine-tune learns exactly what your data file shows it. Get the chat template, the loss mask, and the dedup right, and the run has a chance; get any wrong and no learning rate saves it.

Chat-templated tokens split into masked prompt labels and learned answer labels, so the loss averages over answers only
Chat-templated tokens split into masked prompt labels and learned answer labels, so the loss averages over answers only

Why Does This Exist?

You launch your first SFT job, the loss falls, and the model comes back worse. It rambles, it ignores the roles, it answers questions with more questions. Your first instinct is to tune the learning rate. Don't. You'll usually find the bug in the data file, not the optimizer.

A base model has no taste. It copies whatever your file repeats, including your formatting slips. Train it to predict your prompts and you spend gradient on question writing instead of answering. Show it fifty copies of one answer and it memorizes that answer instead of the pattern behind it. Wrap pairs in the wrong control tokens and it learns a dialect your serving prompt never speaks.

That is what preparation fixes, and it costs far less than another run. Scrubbing buys stability but it also costs coverage: each filter can strip a rare style you wanted to keep. So you clean with a purpose, not on autopilot.

This page covers the four jobs in the file: template, mask, dedup, and an honest split. The loop that consumes the file is supervised fine-tuning. The token contract is chat templates. What narrow data does to old skills is fine-tuning forgetting, and honest measurement is train, validation and test splits. One more boundary: if prompting already gets the behaviour, skip the fine-tune. Prep can't create signal your demonstrations don't contain.

Think of It Like This

A kitchen ticket and the dish that gets judged

You run a kitchen and you train a new cook with order tickets clipped to finished plates. The ticket says what the customer asked. The plate is what you judge. A smart trainer scores only the plate, because scoring the ticket teaches the cook to write tickets instead of cooking.

Your SFT file works the same way. The prompt is the ticket: the model reads it for context. The answer is the plate: the only part the loss should grade. The ticket format is the template, the hidden tickets in the drawer are your test split, and fifty identical practice tickets teach memorization, not cooking.

Where it stops: your cook asks when a ticket is unclear. Your model never asks. It treats a misprinted ticket as truth and learns it perfectly, which is why format slips hurt so much here and so little in a human kitchen.

How It Actually Works

Here's the deal: four jobs, in this order. Template first, then mask, then dedup, then split. Each one assumes your examples show one consistent style you want repeated. Mix grades, tones, or formats in one file and the model learns the mix, because averaging can't pick your favorite.

The one idea you need first

SFT is next-token prediction over demonstrations. The model reads the whole templated pair as context but it should only pay for the answer tokens. That is the whole game of this page, and the full loop is supervised fine-tuning. If that sentence feels new, read that page first, then come back.

Why the template must match exactly

Every chat model expects exact control tokens around each role, and the only source of truth is the tokenizer's own template. Render training pairs and serving prompts with the same call, character for character. A close guess teaches a dialect: low training loss, confused deployment, and a long hunt through hyperparameters that were never the problem. The contract lives in chat templates, so this page won't re-teach it.

What loss masking actually averages

Let NN be the total tokens in one templated example, where a token is one tokenizer chunk. Let PP be the prompt tokens, the part the model only reads. Let AA be the answer tokens, with A=N−PA = N - P. Labels yiy_i decide what each position teaches: yi=−100y_i = -100 means ignore this position, any other yiy_i means predict this token. The loss LL is the mean negative log likelihood over answers only:

L=−1A∑i=P+1Nlog⁡p(xi∣x<i)L = -\frac{1}{A}\sum_{i=P+1}^{N}\log p(x_i \mid x_{<i})

In plain words: for each answer token xix_i, look at the model's predicted probability pp given everything before it, take the log, flip the sign, and average over the AA answers. Prompt positions contribute nothing because their labels sit at −100-100.

Worked example. Take N=9N = 9 tokens with ids [10,11,12,13,14,15,20,21,22][10, 11, 12, 13, 14, 15, 20, 21, 22]. The first P=6P = 6 are the prompt, so their labels become −100-100 and the last A=3A = 3 keep their ids. Say the model's negative log likelihoods on the three answers are 0.70.7, 0.40.4, and 1.11.1. The sum is 0.7+0.4+1.1=2.20.7 + 0.4 + 1.1 = 2.2 and the loss is 2.2/3≈0.7332.2 / 3 \approx 0.733. Skip the mask and the denominator becomes 99 with six prompt terms polluting the average, which is how runs learn to write questions instead of answers.

Why duplicates punch above their weight

Deduplication removes repeats before they buy extra votes. Every copy gets its own gradient, so frequency is weight. Twenty copies in a 200 example file take 20/200=0.1020 / 200 = 0.10, a tenth of every epoch, and the model memorizes that answer instead of the pattern behind it. Kill exact copies first, then near copies that differ by a word or two but teach the same response. The price is real: aggressive filters also eat rare styles, so keep one copy of each pattern instead of zero.

When to split off the test set

Split before you touch anything else. Pull your test slice from the raw pool, then dedup and clean the train split only, and check that no near-duplicate of a test prompt remains in train. Clean first and split later and near-twins land on both sides, so your score measures recall of the cleaning pass instead of new prompts. The full discipline is train, validation and test splits.

Code

The point of this snippet is the check you'll run on every batch: prompt labels are all −100-100 and the loss divides by answers, not total tokens.

def mask_labels(token_ids: list[int], prompt_len: int) -> list[int]:    return [-100] * prompt_len + token_ids[prompt_len:]

def masked_mean_nll(nlls: list[float], prompt_len: int) -> float:    kept = nlls[prompt_len:]    return sum(kept) / len(kept)

ids = [10, 11, 12, 13, 14, 15, 20, 21, 22]print(mask_labels(ids, 6))  # -> [-100, -100, -100, -100, -100, -100, 20, 21, 22]print(round(masked_mean_nll([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.7, 0.4, 1.1], 6), 3))  # -> 0.733

Same nine tokens as the worked example, same 0.7330.733. If your loader prints anything else, the mask is wrong, not the model.

Watch Out For

Training on the prompt

The loss falls and the answers get worse: the model asks questions back, echoes the prompt, or loses the base style. Your denominator counts prompt tokens, so much of the gradient teaches question writing instead of answering.

Fix it by decoding one batch before you launch. Every prompt position must show label −100-100 and the divisor must equal the answer count AA, not NN. In most loaders that means setting labels to −100-100 wherever the prompt mask is true, then asserting the masked count equals PP before the run starts.

Splitting after you clean

Eval scores look great and production disagrees. Near-duplicates straddled the split, so the test measured memory, not generalization.

Fix it by re-splitting from the raw pool first, deduping the train side only, then dropping any train example that closely matches a test prompt. Re-run eval once. If the score drops, the old score was the leak, and the new one is the first honest number you've had.

The Quick Version

  • Fine-tuning data prep turns raw pairs into templated tokens with prompt labels at −100-100, so the loss averages over the A=3A = 3 answer tokens, not all N=9N = 9.
  • The chat template must match the base model's control tokens character for character, at train time and at serve time.
  • Duplicates buy extra gradients, so twenty copies in two hundred examples take a tenth of every epoch: dedup before you train.
  • Split the test set from the raw pool before cleaning, then dedup train only, or twins across the split make your eval lie.
  • If prompting already produces the behaviour, skip the fine-tune: prep can't create signal your demonstrations don't contain.