Fine-Tuning: When You Actually Need It

Read this first: you probably do not need fine-tuning

If you landed here because your LLM feature is not good enough yet, the fix is almost certainly prompting or RAG, not training. Fine-tuning is the last rung of the ladder, not the first: it costs weeks of curating data, it adds a model version you must maintain forever, and it cannot fix missing knowledge. Come back to this article when your prompt already works — and has grown so big and slow that sending it every time hurts.

Why you should care

Fine-tuning is the most glamorous tool in AI engineering and the one most often used for the wrong reason. Knowing when not to do it will save you weeks. The short version: fine-tuning changes how the model behaves, not what it knows — and it makes a good thing cheaper, not a broken thing good. If you are here because your feature is bad, you are on the wrong rung.

What fine-tuning actually is

Fine-tuning means continuing to train a pretrained model on your own examples, so its weights shift toward your task. After tuning, the model does not need your examples in the prompt anymore — the behavior is baked in. It is not teaching the model new facts; it is teaching it a style of behaving: output format, tone, terse domain jargon, consistent structure.

A common confusion: fine-tuning is a bad way to add knowledge. Models fine-tuned on your support docs will still invent product details, because weights are a lossy storage for facts. Knowledge belongs in RAG; behavior belongs in weights.

Words you will see

Word	What it means here
Weights	The billions of numbers inside the model that decide how it behaves. Fine-tuning nudges them.
Pretrained model	A model already trained on a huge amount of text. You start from it instead of from zero.
Fine-tuning	Extra training on your own examples, so the behavior becomes built-in instead of re-explained in every prompt.
RAG	Retrieval: look up your documents at request time and put the facts in the prompt. For knowledge, not behavior.
LoRA	A cheap kind of fine-tuning: instead of updating every weight, train a small add-on layer. Hours instead of days, and you can swap it on and off.
Distillation	Using a big model to generate training examples for a small one — a shortcut to a cheap specialized model.
Drift	Your tuned model quietly getting worse as the world, your data, or the base model changes.
Benchmark	A standard public exam for models. Useful for comparing models, nearly useless for judging your own feature.

Prompting vs RAG vs fine-tuning

The three are not rivals — they solve different problems:

	Prompting	RAG	Fine-tuning
Changes	What you ask, this once	What the model can see	How the model behaves
Good for	Format, tone, constraints	Missing, private or changing facts	A stable style at scale
Cost	Minutes	Days	Weeks, plus a dataset
Reversible?	Instantly	Instantly	No — you now maintain a model version
Goes stale?	No	No — documents are re-read	Yes: retrain when things change

Why you need it later than you think

Almost every team reaches for fine-tuning too early, because it sounds like the “real” engineering solution. In practice the escalation ladder has rungs in front of it:

Better prompting — clearer instructions, examples, output structure. Solves a shocking share of “the model can’t do X” cases.
Few-shot prompting — put 3-5 solved examples in the prompt. The poor-man’s fine-tuning; costs tokens, but changes nothing permanent.
RAG — if the problem is missing or private knowledge, retrieval beats tuning every time.
Fine-tuning — only when behavior is already right but annoyingly expensive or slow to express in prompts: prompts grown to thousands of tokens of pure examples, latency and cost that hurt at scale.

The rule of thumb

If you cannot get the task right with prompting and RAG, you are not ready to fine-tune — you probably do not yet know what you want. Teams that fine-tune successfully almost always do it after months of prompt iteration, with a dataset built from production traces and a golden set from their evals. The evaluation discipline from this section is the prerequisite, not an afterthought.

In one sentence: fine-tuning makes a good thing cheaper, not a broken thing good.

The decision path

Walk these rungs in order and stop at the first one that solves your problem. Most teams never get past the second:

Prompting → RAG → fine-tuning: stop at the first rung that works
→→→→
1. Prompt harder
Rewrite instructions, add structure, add constraints.
Cheapest fix, works surprisingly often. If quality jumps — STOP. You are done; keep iterating here.
out› cost: hours
1/5

What it costs

The bill has three parts, and only the first is visible:

Money — training runs from a few dollars (small open models on your own data) to thousands (frontier models, big datasets). Per-token training prices make small experiments cheap; the trap is the dataset.
Data — you need hundreds to thousands of high-quality input-output pairs. Collecting, cleaning and curating them is weeks of work — usually the real cost.
Lock-in of effort — a fine-tuned model is a model version to maintain: retrain on provider updates, re-run evals, watch for drift. Prompting has none of this maintenance tail.

One honest alternative to know by name: distillation — using a big model to generate training data for a small one. It is how many teams get a cheap, fast, specialized model without hand-writing thousands of examples. Worth a search when fine-tuning starts looking attractive.

Try it: an honest audit before you train

Paste this into your agent or chat, together with your current prompt. It will not flatter you:

Copy this prompt: should I actually fine-tune?

Here is my current prompt and what my LLM feature does. I am considering
fine-tuning. Be skeptical and talk me out of it if I am wrong.

MY PROMPT:
[PASTE YOUR PROMPT HERE]

WHAT IT DOES:
[one paragraph: what the feature is, roughly how many requests per day]

WHAT ANNOYS ME:
[too slow / too expensive / wrong format / wrong facts / inconsistent tone]

Answer, in this order:
1. Is my problem missing KNOWLEDGE or wrong BEHAVIOR? Say which, and quote
 the line of my prompt that makes you think so. If it is knowledge, stop
 here and tell me what to put in retrieval instead.
2. How many tokens is my prompt, and how many are examples that never
 change? If that number is small, say plainly that fine-tuning is not
 worth it yet.
3. Do I have hundreds of real, reviewed input-output examples? If not, say
 what collecting them would take.
4. Give me a verdict: not yet / maybe / yes — and the ONE thing I should try
 first instead.

Do not tell me fine-tuning is a good idea just because I asked.

Check yourself

Your assistant gives wrong answers about your product because it has never seen your documentation. What should you do?

When is fine-tuning the right move?

What is usually the real cost of fine-tuning?

How do you know the fine-tuned model was worth it?

Your takeaway

Before you spend a week on training data, walk the ladder in order and stop at the first rung that solves the problem:

Rewrite the prompt. Hours. Works more often than anyone expects.
Add retrieval if the problem is facts. Days. Most teams stop here — permanently, and happily.
Try few-shot — five solved examples in the prompt — before paying for training.
Fine-tune only when the behavior is already correct and the prompt carrying it has become huge, slow and expensive at scale. Consider LoRA to keep the bill small.
Prove it with evals. Same golden set, same evals: the tuned model must beat the prompted one. If it does not, you bought maintenance for nothing.

One sentence to keep: fine-tuning makes a good thing cheaper, not a broken thing good.