A support team we'd recognise the shape of — most teams running RAG in production would — spent two weeks rewriting their system prompt because the assistant kept answering pricing questions with numbers from a superseded rate card. The prompt was already fine. The rate card was still in the index, sitting next to the current one, and nothing about embedding similarity told the retriever which document was authoritative. The model wasn't confused about instructions. It was confidently reporting the contents of a document it should never have been handed.
This is the pattern behind almost every "the AI is making things up" complaint once the RAG applications work is actually live: the model is not hallucinating in the sense people mean. It is accurately summarising a passage that retrieval should never have surfaced. Fixing that by editing the prompt is like debugging a broken query by changing the font on the results page — the failure happened a step earlier, and no amount of instruction-tuning the generation step reaches back to fix it.
This article assumes you've already decided RAG is the right architecture for the problem — if you're still weighing that against fine-tuning or a plain prompted model, that decision belongs in a separate conversation. What follows is about a RAG system that exists, is retrieving something, and is nonetheless wrong more often than it should be.
Chunking is a precision/context trade, not a solved default
Every RAG pipeline starts with a chunking decision, and most start with the same one: split documents into fixed-size windows, embed each window, index it. It works well enough to demo and badly enough to erode trust once real questions arrive, because the right chunk size depends on what the chunk needs to survive on its own.
Chunks that are too large lose precision. A 2,000-token chunk covering an entire policy section gets retrieved correctly — it is, after all, topically relevant — but the model still has to find the one sentence that answers the question inside a block of text that also contains four adjacent, similar-sounding rules. The retrieval step did its job; the generation step now has to do disambiguation the retrieval step should have done, and it doesn't always get it right.
Chunks that are too small lose context. Split a table from its header, or a clause from the definition it depends on, and you get a chunk that embeds well and means nothing on its own. A line reading "the reduced rate applies for the first 90 days" is a different fact depending on which plan's rate the preceding sentence was describing, and if that sentence lives in a different chunk, the retriever has no way to know they belong together.
The fix is not a universal chunk size — it's chunking along the document's actual structure and carrying enough breadcrumb context into each piece that it stands alone:
# Fixed windows ignore what the document actually is
chunks = [text[i:i+1000] for i in range(0, len(text), 800)]
# Structure-aware chunking carries the context a reader would need
for section in parse_sections(document):
for chunk in split_by_paragraph(section.body, max_tokens=400):
index(
text=f"{document.title} > {section.heading}\n\n{chunk}",
metadata={"doc_id": document.id, "section": section.heading},
)
There's no single right chunk size across a heterogeneous corpus — a legal clause, a support ticket thread, and a product spec table all fail differently at the same fixed window. Chunking strategy is a modelling decision made per document type, and it deserves the same scrutiny as the embedding model choice, not the leftover five minutes at the end of a sprint.
The retriever finds "similar," not "true"
Embedding-based retrieval ranks by semantic similarity, and semantic similarity is not the same property as factual relevance. A query about your refund window can retrieve a passage about a different company's refund policy that happens to use nearly identical phrasing, or last year's terms sitting a few cosine-similarity points away from this year's. The vector index has no concept of "current" or "correct" — it has a concept of "sounds like this."
This mismatch shows up in a few recognisable shapes:
- Near-duplicate documents with different facts. Two versions of the same policy, one superseded, both indexed, both semantically close to the query. Nothing ranks the current one above the old one unless you've told the system which is which.
- Stale documents nobody removed. An onboarding guide from before a product change, still in the index, still perfectly well-written, now wrong. Retrieval has no sense of expiry unless expiry is metadata you actually filter on.
- Adjacent-but-wrong domains. A query about one product line surfacing a passage about a similarly named one, because the embedding space clusters them together on vocabulary, not on which one the user actually asked about.
- Short queries landing in the wrong part of embedding space. A three-word query and a 400-word chunk are structurally different objects, and pure vector similarity between them is often a weaker signal than a hybrid of keyword and vector search.
- No re-ranking step, so the right passage is buried at position seven. Vector search returns an approximately-ranked top-k, not a precisely-ranked one, and if your pipeline hands the raw top-k straight to the model, the passage that actually answers the question can sit below three passages that are merely on-topic. A re-ranking pass over the retrieved set — even a lightweight cross-encoder step before generation — routinely reorders results in ways that change the answer.
None of these are exotic edge cases. They are what happens by default when an index accumulates real organisational content over real time, and none of them announce themselves — the model still returns a fluent, well-formatted answer, because a language model's fluency is independent of whether the material underneath it was the right material.
An index is a data asset, and it needs the housekeeping one implies
Teams that would never let a production database accumulate duplicate, unlabeled, undated records will do exactly that to a vector index, because it doesn't look like a database — it looks like an input to a demo. In practice it needs the same lifecycle discipline: versioning, deduplication, expiry, and a defined path from "source document changed" to "index reflects that."
Retiring a document from the source system does nothing to the index unless something actively removes or re-embeds it. The most common way we see this go wrong is quietly: a document is updated in the CMS, the corresponding index entry is never refreshed, and the system now serves two answers to the same question depending on which chunk retrieval happens to favour that day. Reindexing has to be a triggered, monitored step tied to the source of truth changing — not a one-time job run at launch and forgotten.
Retrieval quality has to be measured, not eyeballed
The most common way we see a RAG project go sideways after launch isn't any single failure above — it's that nobody has a way to notice when one is happening. Someone reads five transcripts, judges them "pretty good," and ships. A month later a pattern of wrong answers has accumulated, and it's a shrug rather than a bug report, because there was never a baseline to regress against.
The fix is an evaluation set: a fixed collection of real questions with known-correct source documents, checked before any change to chunking, embeddings, or prompts, and re-run after. Measure retrieval on its own, separately from generation — was the correct source document in the top-k results at all, independent of whether the final answer sounds right — because a good model answering from a bad retrieval set will produce a confident, readable, wrong answer every time, and no amount of prompt polish moves that number. This is exactly the kind of groundwork worth scoping properly before a single line of retrieval code gets written, which is why it belongs in an upfront AI consultancy engagement rather than as an afterthought once users start complaining.
Once you have that measurement in place, showing users the source alongside the answer is the second-cheapest reliability improvement available. Citation display doesn't just build trust — it gives every user a lightweight, distributed verification step that catches retrieval failures your eval set didn't anticipate, because real usage always finds questions your test set didn't.
The permission problem is the same one, wearing a security hat
Everything above assumes the retriever is allowed to search everything it's searching. That assumption is where retrieval quality and security stop being separate topics. A RAG index built over documents with different access levels — HR records, contracts, one customer's data next to another's — has to enforce the same access control the source systems enforce, applied as a filter at query time, derived from the authenticated caller's identity rather than anything the query or the model can influence. Skip that, and the system isn't producing a wrong answer — it's leaking a right one to someone who shouldn't have received it, which is the specific way this failure mode turns from a quality problem into an incident. It's the same concern we've written about for agents that read from a permissioned RAG layer and then act on what they find: the retrieval layer is the enforcement point, and there isn't a later stage where a missed permission check gets caught.
Where this lands
Most "the AI is wrong" reports are retrieval reports in disguise, and treating them as prompt problems burns time without moving the number that matters. If your system is giving confidently wrong answers, look first at what actually made it into the context: was the right document chunked in a way that preserved its meaning, was it still current, was it ranked above the plausible-sounding distractor next to it, and was it something this particular user was even allowed to see. Retrieval quality isn't a property you get right once at launch — it degrades quietly as your document set grows, changes, and accumulates duplicates, which makes it an operational concern with a measurement and a maintenance cadence, not a checkbox on a build plan. If you're running a RAG system that's giving you this kind of trouble, or scoping one and want the retrieval and permission questions answered before the build starts rather than after a bad month, that's a conversation we're glad to have.