RAG or fine-tuning: what to use for company knowledge
This question comes up on most technical calls and is almost always framed wrongly, because it assumes a choice between two alternatives. In fact RAG and fine-tuning solve different problems and rarely compete in practice. Below we separate the two problems so you can tell which one you actually have — and why, in company implementations, the answer in eight cases out of ten is "RAG, and that's enough."
- RAG answers "how is the model supposed to know this"; fine-tuning answers "how is the model supposed to behave."
- Knowledge that changes — price lists, procedures, stock levels — belongs in RAG, because updating it means swapping a document.
- In most company implementations fine-tuning isn't needed at all; when it is, it is added to RAG rather than replacing it.
Two different problems
RAG (retrieval-augmented generation) solves the knowledge problem: on its own the model doesn't know your price list, your complaints procedure or a particular customer's commercial terms. RAG supplies them at the moment of answering — it retrieves the right fragments of documents and passes them in as context. The model doesn't "learn" them permanently; it gets them each time, like a note to consult.
Fine-tuning solves the behaviour problem: how the model should phrase answers, what tone to use, what format to return results in, how to react to unusual wording. Training changes the model itself based on examples. It does not add facts in a way you can rely on — and that is the most common misunderstanding and the most expensive mistake in this category.
Why RAG usually wins inside companies
Company knowledge has one property that settles the matter: it changes. A price list holds for a quarter, a procedure gets a new clause after an audit, a customer's commercial terms change after a negotiation. In RAG an update means swapping a document in the index — minutes. With fine-tuning every such change means another training run, and in between the model asserts outdated facts with complete confidence.
Currency. A change in a document takes effect immediately and applies to every answer. There is no window in which the system still knows the old version of a procedure.
Verifiability. An answer can be tied back to its source — a specific document and a specific passage. With fine-tuning there is nothing to point at, because the knowledge has dissolved into the model's weights.
Access control. Permissions can be applied at the retrieval stage: a salesperson sees their own terms, not every customer's. A model trained on everything can no longer be pulled apart.
Cost of change. Adding a new area of knowledge means indexing more documents, not another cycle of training and evaluation.
When fine-tuning does make sense
There are situations where adding context isn't enough. If answers have to hold a very specific format or industry jargon the model consistently misses, and you have several hundred good historical examples, training can settle it more cheaply than ever-longer instructions. The same goes for very narrow classification tasks at high volume, where a smaller trained model can be cheaper to run than a large general one.
Even then, fine-tuning is added to RAG rather than replacing it: a trained model still needs current facts supplied in context. The order matters too — first we launch RAG and measure quality, and only if something is still missing after the retrieval and the instructions have been tuned do we consider training. In practice, in most implementations that second step never happens.
The cost of updating knowledge in each approach. This is the one difference you need to understand to make the right decision — every other argument follows from it.
Frequently asked questions
Doesn't fine-tuning give more accurate answers?
It gives a more predictable form of answer, not greater fidelity to the facts. If the problem is that the model muddles details from your documents, training won't fix it — better retrieval and a cited source will.
Does RAG mean sending our whole knowledge base to the model?
No. Only the fragments needed for a single answer reach the model, usually a few paragraphs. The base itself stays on your side or in a chosen EU region, and permissions limit what can be retrieved in the first place.
Where do we start if we don't know which one we need?
With the question of whether your knowledge changes and whether answers have to be backed by a source. Two yeses mean RAG. Fine-tuning waits for later, and only if measurement on a running system shows something is missing.