Why RAG answers badly: the six most common causes
"The system is hallucinating" is the most frequent diagnosis and the least often correct one. In RAG systems a wrong answer almost always has an earlier cause: the model was handed the wrong passages and described them faithfully. So the fix starts with retrieval, not with the instructions given to the model. Below are six places where quality breaks down, ordered from the most common, and a simple way to work out which one you are dealing with.
- In most cases the model was given the wrong passages, not asked to interpret good ones badly.
- The most common cause is out-of-date documents in the index — the system correctly quotes a version that no longer applies.
- Diagnosis starts by checking what the system retrieved, before anything in the prompts is touched.
Before you blame the model
A RAG system answers in two steps: first it retrieves document passages matching the question, then it composes an answer from them. If step one returns the wrong material, step two has nothing to build a good answer from — and it will usually build a plausible-sounding bad one. That is why the first diagnostic move is always the same: check which passages the system pulled, before you change anything in the instructions.
Six causes, starting with the most common
Out-of-date documents in the index. A price list from two quarters ago sits next to the current one and nothing distinguishes them. The system quotes correctly, just not the right thing. Fix: removing old versions from the index is part of the update process, not an annual tidy-up.
Contradictory sources. The procedure says one thing, the meeting note another, the email to the team a third. With no agreed hierarchy of sources, the system picks whichever happens to match the wording of the question better. Fix: a hierarchy written into the metadata, not left to guesswork.
Badly chunked documents. A passage ends halfway through a table of terms, or separates an exception from the rule it applies to. The model gets half a sentence and completes the rest. Fix: chunk along the structure of the document, not on a fixed character count.
Questions phrased differently than the documents. The customer writes "returning goods", the procedure says "quantity complaint". Semantic search softens this, but with industry jargon it does not disappear. Fix: a dictionary of company synonyms and query expansion before retrieval.
Knowledge that simply is not there. The answer exists only in the head of the person who has run that topic for years. The system has nowhere to get it from, so it constructs something plausible. Fix: this is not a technical problem — the missing document has to be written.
No instruction to say "I don't know". If the system has no explicit threshold below which it declines to answer and escalates, it will always answer. Fix: a confidence threshold and escalation as part of the rules, agreed with the team.
A change of diagnosis that sorts out most reports of poor answer quality. Until you know what the system retrieved, every change to the model's instructions is guesswork.
How to separate these causes in practice
Take twenty real bad answers — not invented test cases — and for each one write down two things: which passages the system retrieved, and whether the correct answer was among them at all. That splits everything into two piles. If the right passage was not retrieved, the problem is in the index, the chunking or the query — causes one through four. If it was retrieved and the answer is still wrong, only then is it worth looking at the instructions and the confidence threshold.
That split takes half a day and saves weeks of work in the wrong place. In audits of existing RAG systems we take over, the first pile is usually clearly larger — and inside it, cause number one dominates: documents nobody removed after an update.
Frequently asked questions
Will a better model solve this?
Partly, and only for the cases in the second pile. If the right passage never made it into the context, no model will guess it — and a stronger one will phrase the wrong answer more convincingly, which makes things worse, because it is harder to catch.
How often does the index need refreshing?
As often as the knowledge changes. In practice this is wired into the existing document workflow: publishing a new version of a procedure updates the index and withdraws the previous one. Refreshing manually "every now and then" is the source of cause number one.
Should the system cite the source of its answer?
Yes, for two reasons. The user can verify the answer in two seconds instead of assuming it is correct. And you get a diagnostic tool: a report of a bad answer arrives with the document the system considered relevant already identified.