RAG or fine-tuning?
This is the most common technical question we are asked, and the answer is usually retrieval. Not always — but the exceptions are narrower than most teams assume.
The short answer
Use retrieval when you need the model to know facts: your products, policies, documentation, customer records. Facts change, and retrieval means updating a document rather than retraining anything. It also gives you citations, which is what makes an answer verifiable.
Use fine-tuning when you need to change behaviour rather than knowledge: a consistent output format, a specialised tone, or a narrow classification task where a smaller cheaper model can be taught to match a larger one. It teaches a model how to respond, not what is true.
- Retrieval for facts, fine-tuning for behaviour
- Retrieval updates in minutes; fine-tuning needs a retraining cycle
- Retrieval gives citations; fine-tuning cannot
- Fine-tuning can cut cost by making a small model competent
- Most production systems that fine-tune also use retrieval
- Start with retrieval and good prompting; escalate only if measured
What each one actually changes
Retrieval changes what the model can see. You keep your content in a store, fetch the relevant parts at question time, and put them in the prompt. The model's knowledge is unchanged; its context is not.
Fine-tuning changes how the model behaves. You train on examples until it adopts a format, a tone, or a classification habit. It does not reliably teach facts — a fine-tuned model will still invent a policy number it was trained on, because training adjusts tendencies rather than installing a lookup table.
That distinction settles most cases. If your problem is 'the model does not know our stuff', that is retrieval. If it is 'the model knows, but answers in the wrong shape', that is fine-tuning.
Why retrieval usually wins on operations
Your content changes. With retrieval, updating a document updates the answers within minutes and the change is auditable. With fine-tuning, it means assembling a new training set and running the job again, and you cannot point at which example caused a given answer.
Retrieval also gives you citations, which is often the requirement that actually matters. In a regulated setting, an answer that cannot show its source is not usable regardless of whether it is correct.
And it fails more gracefully. When retrieval finds nothing relevant, a well-built system says so. A fine-tuned model with no relevant knowledge produces something fluent and wrong, which is considerably more dangerous.
When fine-tuning is the right call
Rigid output formats that prompting keeps drifting away from. High-volume classification where a smaller fine-tuned model is far cheaper per call than a large general one. A distinctive voice that no amount of prompt instruction reproduces. Or latency budgets that a smaller specialised model can meet and a larger one cannot.
In practice the strongest systems often use both: retrieval supplies the facts, and a light fine-tune fixes the shape of the response. Deciding between them as though they were rivals is usually the wrong framing.
Frequently asked questions
Can we do both?
Yes, and mature systems often do: fine-tune a small model for format and tone, then feed it retrieved facts at question time. Start with retrieval, prove the need for the rest.
Will fine-tuning stop hallucination?
No — this is the most expensive misconception in the field. Fine-tuning on your documents teaches style and pattern, not reliable recall. It can make a model hallucinate more confidently. Retrieval plus citations is the mitigation.
How much data does fine-tuning need?
Fewer examples than people expect for behaviour tasks — often several hundred good ones. The difficulty is that they must be consistent, and inconsistent examples actively make things worse.
What about very long context windows instead?
Useful, but not a replacement at scale: cost grows with every token, latency suffers, and accuracy on details buried in the middle degrades. Retrieval sends only what is relevant.
Tell us what you are building.
Send a short description of the problem and we will reply within one business day with an honest view of scope, cost and whether we are the right person for it.
Or email directly: contact@hire-ai-dev.com