When to Fine-Tune

Why Does This Exist?

You are building a support helper that routes tickets and drafts replies in your company's voice: same greeting, same structure, same way of citing policy. Your first version is a prompt with standing rules plus three tickets paired with replies you like. You can rewrite it in minutes and ship again before lunch.

Then the misses start, and someone says the reflexive line: let's fine-tune it. That jump is usually early, and it costs you in a specific way. Fine-tuning buys a built-in habit at the price of a data pipeline, training runs, evals, and re-tuning every time the habit should change. If the real cause was a vague instruction or a missing policy page, you have paid that price and kept the bug, because the bug was never in the weights.

This page is the decision, not the how. When the ask is unclear you want prompt anatomy. When facts are missing or change you want RAG architecture. Once you decide the answer is yes, you want supervised fine-tuning for the training step and fine-tuning data preparation for the dataset. Here you learn which rung your failure lives on.

The one idea you need first. A prompt tells the model what to do, retrieval hands it facts at request time, and fine-tuning rewrites weights so a behavior sticks without asking. Mix up does not know with does not act right and you will climb the wrong rung.

Think of It Like This

How It Actually Works

You've got three levers, cheapest first, and each one changes a different thing. Climb only when the current rung has a failure it cannot fix by construction.

The three rungs and what each one changes

Prompt engineering sharpens the ask: clearer rules, tighter format, a few demonstrations of the ticket and reply pattern you want. Nothing about the model changes. You iterate in minutes, which is why you start here.

Context and retrieval hands the model facts at request time instead of asking it to remember them. The ticket helper stops guessing the refund window because the current policy page sits inside the prompt. Same model, better evidence. You pay for an index and a fetch, still no retraining.

Fine-tuning changes the weights themselves. Let $W$ mean the model's weights, the numbers training learned. Fine-tuning keeps training $W$ on your demonstrations so a voice or format becomes built in rather than requested each time. That's the trade-off in one line: you buy behavior that survives short prompts, and you pay with curated data, compute, evals, and a pipeline you must rerun whenever the target shifts. Do not buy it for facts that change next quarter.

The test that earns the climb

You have earned the top rung when four things are true at once. Name them before you spend.

• The miss survives rewording. Let $k$ mean the number of distinct prompt rewrites you honestly tried, with explicit format rules and examples. If $k = 1$, you haven't tested the cheap rung. Try at least three genuinely different asks.
• It is behavior, not knowledge. Let $E_{\text{fact}}$ mean replies with wrong or outdated facts, and let $E_{\text{style}}$ mean replies with right facts but wrong voice or format. Fine-tuning is for $E_{\text{style}}$ that stays high while $E_{\text{fact}}$ falls.
• You have the demos. A handful of rushed examples is not a training set. You need consistent pairs in the target voice, cleaned and deduped, plus a held-out sample you grade by hand.
• The volume pays for the pipe. A helper answering hundreds of tickets a day can carry curation and re-tuning. A tool three teammates use each week usually cannot.

The assumption underneath: your target behavior is stable, gradeable, and worth freezing into $W$. When that assumption fails, when the voice still shifts weekly or no two graders agree, the run teaches noise and you feel it as regressions elsewhere.

Worked example: 50 replies, counted twice

Take $N = 50$ of your own tickets, where $N$ means the total replies you grade. Grade each reply twice: facts right or not, voice right or not.

With your best prompt alone you count $E_{\text{fact}} = 15$ and $E_{\text{style}} = 12$. As rates that is $15 / 50 = 0.30$ fact errors and $12 / 50 = 0.24$ style errors. So 30 percent wrong facts, 24 percent wrong voice. Two problems at once.

Now add retrieval with the current policy page and change nothing else. You recount the same 50. Eleven fact errors vanish because the page was there to copy, so $E_{\text{fact}}$ drops from 15 to 4:

Facts fall to 8 percent. Style barely moves: one reply tips over, so $E_{\text{style}}$ drops from 12 to 11:

Style sits at 22 percent across $k = 3$ rewrites, with facts now mostly right. That split is the whole decision. Retrieval fixed knowledge. The leftover is habit. Fine-tuning is now defensible, and only for that leftover.

Code

This router turns the failure you see into the rung you try next. Read the two prints as the two cases from the example.

1from dataclasses import dataclass2
3
4@dataclass5class Failure:6    facts_wrong: bool7    unclear_ask: bool8    bad_style: bool9    persists_many_prompts: bool10
11
12def next_rung(f: Failure) -> str:13    if f.unclear_ask:14        return "prompt engineering"15    if f.facts_wrong:16        return "context / retrieval"17    if f.bad_style and f.persists_many_prompts:18        return "fine-tuning"19    return "prompt engineering"20
21
22print(next_rung(Failure(False, False, True, True)))  # -> fine-tuning23print(next_rung(Failure(True, False, False, False)))  # -> context / retrieval

Watch Out For

The Quick Version

• Prompting fixes unclear asks, retrieval fixes missing or changing facts, and fine-tuning fixes behavior that survives both.
• A sharper sentence or one more retrieved page means the cheap rung still has room, so stay there.
• Fine-tuning rewrites weights, which buys built-in voice at the price of data work, evals, and re-tuning on every shift.
• Changing facts belong in retrieval against the live document, since weights bake in a snapshot that goes stale.
• The climb is earned by proof: the same style miss across several rewrites, right facts with wrong voice, clean demos to train on, and enough volume to carry the pipeline.

What to Read Next

• Supervised Fine-Tuning
• Fine-Tuning Data Preparation
• RAG Architecture
• Prompt Anatomy