Reach for Retrieval Before You Reach for Fine-Tuning

A common pattern: an LLM gives generic answers about your product, so someone proposes fine-tuning it on your docs. It sounds right — teach the model your world. But fine-tuning is often the slowest, most brittle way to solve what is really a knowledge problem, not a behavior problem.

Most 'it doesn't know our stuff' problems are retrieval problems

Fine-tuning is good at shaping how a model responds — tone, format, structure. It's a poor and costly way to inject facts, because facts change and retraining is slow. Retrieval-augmented generation puts the right context in front of the model at request time, so updating knowledge is as easy as updating a document.

• Knowledge that changes (docs, policies, catalogs) → retrieval, not fine-tuning.
• Consistent format or style the model keeps missing → consider fine-tuning.
• Both? Retrieve for facts, fine-tune for behavior — they're complementary, not rivals.

Get retrieval right before blaming the model

When a RAG system gives bad answers, the model is usually the last thing at fault. The failure is almost always upstream: poor chunking, weak retrieval, or context that doesn't actually contain the answer. Fix the pipeline before you reach for a bigger or fine-tuned model.

Before fine-tuning, ask:
  1. Did retrieval return the right documents?   (measure recall@k)
  2. Was the answer actually in the context?     (read the prompt)
  3. Did chunking split the answer in half?      (inspect chunks)
Most 'model' failures die at step 1 or 2.

When fine-tuning does pay off

Fine-tuning earns its cost when you need a consistent behavior at scale that prompting can't reliably produce, when you want a smaller, cheaper model to match a larger one on a narrow task, or when latency budgets rule out stuffing long context. In those cases it's a precision tool — just not the first one to reach for.