Gen AI

When to Fine-Tune

Fine-tuning is the priciest rung on a ladder starting with sharper prompts, then retrieved facts, and only reshapes the model's weights once both genuinely fail.

Prompt engineering, retrieval, and fine-tuning form a ladder of rising cost and slower iteration, and you only climb a rung once the one below it has a specific failure it cannot fix

Why Does This Exist?

Say you're building a support-ticket classifier that also drafts the reply. It has to route the ticket correctly and answer in the company's exact voice — a specific greeting, structure, and way of citing policy. Version one is just a prompt: a system message describing the tone, plus a few examples of tickets paired with the replies you want. Cheap to write, cheap to change — edit a paragraph and redeploy in minutes.

The trouble starts when the prompt keeps missing, and the reflexive fix is often "fine-tune it." That's usually premature. Fine-tuning earns its place, but it sits atop a ladder with two cheaper rungs beneath it, and most failures that look like they need fine-tuning get fixed by climbing one of those first. Jump straight to fine-tuning and you've bought retraining runs and an ongoing pipeline for a problem a sharper sentence or a retrieved policy page would have solved for a fraction of the cost — and you still haven't fixed it, because the cause was never the model's weights.

Think of It Like This

Coaching a new hire, not sending them back to school

A new support rep writes replies in the wrong voice. First you hand them sharper instructions — a style guide, two model replies to copy. If they keep citing last year's refund window, you hand them the policy binder so they stop guessing at facts they were never given. But if they still write clipped and terse when the brand wants warm and specific, no matter how you phrase the memo, you don't send another memo. You put them through weeks of coaching until the new voice becomes automatic.

That coaching is fine-tuning: slow, expensive, and reserved for a habit plain instructions can't touch.

How It Actually Works

The ladder, in order, and why the order matters

Three rungs, cheapest first. Prompt engineering is a paragraph and a few examples — few-shot demonstrations of the ticket-and-reply pattern you want. It costs minutes, and you can rewrite it as many times as you like. Context and retrieval comes next: when the model gets facts wrong, you stop asking it to remember policy and hand it the actual document at request time, the same closed-book-to-open-book move RAG architecture covers directly. That costs an indexing pipeline, more than a prompt edit but still no retraining. Fine-tuning is last: you change the model's weights so a behavior becomes built in rather than requested.

The order isn't arbitrary. Each rung costs more to build and iterates slower than the one before it, so you only climb when the current rung has a specific, name-able failure it structurally cannot fix. Climbing early means paying fine-tuning's price for a bug a cheaper rung would have closed on the next deploy.

What each rung is actually good at fixing

Each rung answers a genuinely different question, and matching the fix to the actual cause is the whole game. Prompt engineering fixes "the instructions weren't clear or specific enough" — the model had everything it needed and simply wasn't told precisely enough what to do with it. Retrieval fixes "the model doesn't have access to a fact, or the fact changes over time" — the ticket classifier citing an outdated refund window isn't a phrasing problem, it's a missing-document problem, and no amount of prompt polish teaches a model something it was never shown. Fine-tuning fixes something different again: "the model can't reliably produce a certain style, format, or behavior no matter how it's asked."

That clause is the diagnostic test. If a better-worded instruction or one more retrieved paragraph would obviously fix the failure, you're not at the fine-tuning rung yet — you're at a cheaper one you haven't finished climbing.

Exit criteria for actually justifying the climb

A team has exhausted the cheaper rungs, and fine-tuning is justified, when several things are simultaneously true. The failure persists across many different prompt phrasings — not one prompt that didn't work, but a dozen honest attempts, including explicit formatting instructions and few-shot examples, that all land on the same wrong tone or structure. It's demonstrably a behavior or format issue rather than a facts issue — the ticket responder gets the policy right and the routing right, and still writes in the wrong voice. There's enough quality demonstration data to fine-tune on — a handful of hand-picked examples isn't a training set. And the cost of building and maintaining a fine-tuning pipeline — data curation, training runs, evaluation, re-tuning when the desired style shifts — is justified by how much volume runs through the system. A ticket responder handling thousands of tickets a day clears that bar easily; a tool three people use internally usually doesn't.

Show Me the Code

A minimal ladder-router: describe the failure, get back the next rung to try.

1from dataclasses import dataclass2
3@dataclass4class Failure:5    facts_wrong: bool          # model states an incorrect or outdated fact6    unclear_ask: bool          # instructions were vague or under-specified7    bad_style_or_format: bool  # correct facts, still wrong tone/structure8    persists_many_prompts: bool9
10def next_rung(f: Failure) -> str:11    if f.unclear_ask:12        return "prompt engineering"    # sharpen the instructions first13    if f.facts_wrong:14        return "context / retrieval"   # inject the current facts15    if f.bad_style_or_format and f.persists_many_prompts:16        return "fine-tuning"           # behavior, not facts, survives rewording17    return "prompt engineering"        # default: cheapest rung first18
19print(next_rung(Failure(False, False, True, True)))  # -> fine-tuning

Watch Out For

Fine-tuning a facts problem that retrieval would have solved

The ticket responder cites a refund window that changed last quarter, and the team reads that as "the model needs retraining." It doesn't — it needs the current policy document in front of it at request time. Fine-tuning on updated facts bakes in a snapshot that goes stale the next time policy changes, so you're back to retraining on a schedule instead of pointing retrieval at a document that updates itself.

Fine-tuning on too little or too low-quality demonstration data

A team collects thirty example replies, most written in a hurry, and fine-tunes on them expecting a consistent house style to emerge. Thirty inconsistent examples teach thirty inconsistent lessons, and the result often writes worse than the base model with a good prompt did. Fine-tuning amplifies whatever is in the data — it doesn't average sloppiness into polish.

The Quick Version

Three rungs, rising cost and falling iteration speed: prompt engineering, then context/retrieval, then fine-tuning.
Prompt engineering fixes unclear instructions. Retrieval fixes missing or changing facts. Fine-tuning fixes a style, format, or behavior the model can't hit no matter how it's asked.
The diagnostic test: if a sharper prompt or one more retrieved paragraph would obviously fix it, you haven't earned fine-tuning yet.
Exit criteria: the failure survives many prompt rewrites, it's a behavior issue rather than a facts issue, there's real demonstration data to train on, and the volume justifies a training pipeline.
Climbing early doesn't just waste money — it usually doesn't even fix the bug, because the cause was never in the weights.