Here is the short version, and it holds up more often than not: if the complaint is "the model doesn't know our stuff", you have a retrieval problem. If the complaint is "the model knows the answer but won't say it the way we need", you have a prompting-then-fine-tuning problem. Almost every muddled month we've seen a team lose to this question came from mixing those two up — usually by fine-tuning a model on a corpus of internal documents in the hope that facts would stick, and then discovering that the model had learned the shape of the documents without reliably learning their contents.
That distinction — what the model knows versus how the model behaves — is the whole framework. Everything below is a consequence of it.
The two levers do different jobs
Retrieval-augmented generation (RAG) puts facts into the context window at request time. The model is unchanged; you are changing its input. That means the knowledge can update the moment your source of truth updates, you can show the user exactly which document produced a claim, and you can filter what gets retrieved per user.
Fine-tuning adjusts weights so the model's default behaviour shifts. It is excellent at teaching task-shape: always return this JSON schema, always classify into these seven categories, adopt this register, follow this multi-step reasoning pattern for this narrow domain. It is poor at installing a large body of specific, changeable facts, and worse at letting you revoke one of them later.
The revocation point deserves emphasis because it is where the two approaches diverge most sharply in practice. If a document is retracted, superseded, or turns out to have been confidential, RAG handles it with a delete from an index. Fine-tuning handles it by retraining. If your knowledge has any compliance dimension at all — a right-to-erasure request, a document that must stop being cited on a given date — that alone often settles the architecture.
When to use RAG
Reach for retrieval when any of these are true:
- The knowledge changes. Pricing, policies, product documentation, ticket history, regulations. Anything with a version number or an effective date.
- Answers must be cited. If a user needs to click through to the source paragraph — and in regulated or high-trust contexts they usually do — you need the retrieved chunk to survive into the response as a reference.
- Access is per-user. Two employees asking the same question must get answers drawn from different document sets. This is not achievable with baked-in weights.
- The corpus is large or long-tail. You cannot fine-tune your way through hundreds of thousands of documents where any one of them might matter for a given query.
When to fine-tune
Reach for fine-tuning when the model already has the capability but not the discipline:
- Structured output that must be reliable. You have prompted for a JSON schema, and it is correct most of the time — but "most of the time" fails at volume.
- Domain register and tone. Legal summaries, clinical notes, incident reports. These have conventions that are tedious to specify in a prompt and cheap to demonstrate in examples.
- A narrow, repeated classification or extraction task. Where a smaller fine-tuned model can match a much larger prompted one at a fraction of the latency and cost.
- Prompt length has become the bottleneck. When your system prompt has grown into a style guide, distilling it into weights buys back context and tokens.
Our sequencing rule is simple: prompt first, then evaluate, then fine-tune. Fine-tuning before you have a fixed evaluation set is spending money to change behaviour you cannot measure. And a surprising number of "we need to fine-tune" conversations end once someone writes six good few-shot examples and a stricter output constraint.
The decision table
| Requirement | RAG | Fine-tuning | Notes | | ---------------------------------------- | --- | ----------- | --------------------------------------------- | | Facts change weekly or faster | ✅ | ❌ | Retraining cadence cannot keep up | | Must cite sources | ✅ | ❌ | Weights carry no provenance | | Per-user permission scoping | ✅ | ❌ | Filter at retrieval; weights are global | | Must support deletion / erasure | ✅ | ❌ | Delete from index vs. retrain | | Enforce a strict output schema | ⚠️ | ✅ | Try constrained decoding first | | Domain tone and formatting conventions | ❌ | ✅ | Demonstrate, don't describe | | Reduce latency and cost on a narrow task | ❌ | ✅ | Smaller specialised model wins | | Large, long-tail corpus | ✅ | ❌ | Too much to memorise | | Shrink an overgrown system prompt | ❌ | ✅ | Distil the style guide into weights | | Teach a multi-step domain procedure | ⚠️ | ✅ | Often hybrid: fine-tune shape, retrieve facts |
The hybrid case is common, and it is not a cop-out
The mature architecture for a serious internal assistant is usually both. Fine-tune (or carefully prompt) a model so that it reliably follows your answer format, cites in your house style, declines confidently when retrieval comes back empty, and calls tools with correct arguments. Then feed that model retrieved context at runtime for the actual facts.
The division of labour is clean: weights own the behaviour, retrieval owns the truth. A useful diagnostic when a hybrid system misbehaves is to ask which half failed. If the answer is wrong but well-formed and confidently cited to a document that does not support it, look at retrieval and grounding. If the answer is factually fine but ignores your schema, wanders in tone, or invents a citation format, look at the model's behaviour layer.
Failure modes we see repeatedly
Retrieval quality dominates output quality
This is the one teams underestimate most. In a RAG system, the ceiling on answer quality is set by whether the right chunk made it into the context — not by which frontier model you chose. Garbage retrieval produces a garbage answer from any model, and the better the model, the more fluently it will justify the wrong document. Before anyone proposes a model upgrade to fix accuracy, we want to see retrieval measured on its own: for a set of real questions, was the correct source document in the top-k results at all? If it wasn't, no amount of model spend will help.
Chunking that destroys context
Fixed-size splitting is the default and it is frequently wrong. Cutting a table away from its header, separating a clause from the definition it depends on, or splitting a procedure mid-step all produce chunks that are individually retrievable and individually meaningless. Chunk along the document's own structure — sections, clauses, rows — and carry enough parent context into each chunk that it stands alone.
# Not this: fixed windows that cut across meaning
chunks = [text[i:i+1000] for i in range(0, len(text), 800)]
# Closer to this: split on structure, then attach the breadcrumb
for section in parse_sections(document):
for chunk in split_by_paragraph(section.body, max_tokens=500):
index(
text=f"{document.title} > {section.heading}\n\n{chunk}",
metadata={
"doc_id": document.id,
"section": section.heading,
"effective_date": document.effective_date,
"acl": document.allowed_group_ids, # see below
},
)
Embedding-model mismatch
Queries and documents must be embedded by the same model, and if you change that model you must reindex everything. This sounds obvious and it is still one of the most common causes of a system that "worked in the prototype and got worse after we upgraded". A related trap: short user queries and long document chunks occupy different regions of embedding space, which is why query rewriting or hybrid keyword-plus-vector search so often outperforms pure vector similarity.
The permissions leak
This is the failure mode we care about most, because it is a security incident rather than a quality problem. A naive RAG pipeline indexes everything it can read — typically with a service account that has broad access — and then retrieves purely on semantic similarity. The result is a system that will cheerfully surface a salary review, a board memo, or another customer's record to whoever phrased the question well enough. No one attacked anything. The retriever simply did its job against an index that had no concept of who was asking.
The fix is permission-aware retrieval: carry the source document's access-control list into the index as metadata, and apply it as a hard pre-filter on every query, derived from the authenticated user's identity — never from anything the user or the model can influence.
-- Filter before ranking, not after. Post-filtering a top-k result set
-- silently degrades recall and still requires the ACL check anyway.
SELECT chunk_text, doc_id
FROM document_chunks
WHERE acl_group_id = ANY(:caller_group_ids) -- from the session, not the prompt
AND effective_date <= now()
ORDER BY embedding <=> :query_embedding
LIMIT 8;
Two related habits matter as much as the filter itself. First, treat the index as a copy of your data with the same classification as the original — if the source documents contain PII, so does your vector store, and it inherits the same retention, encryption, and erasure obligations. Second, remember that retrieved text is untrusted input. A document containing instructions aimed at the model is a live injection vector, and it becomes a privilege escalation the moment your assistant can also call tools.
Where we come in
We build retrieval and fine-tuning pipelines for teams who need them to work under real conditions — and because we also do security and compliance work, we tend to ask about the access-control model and the PII path in the first design conversation rather than the last. That is not a different service bolted on; permission-aware retrieval is simply what a correctly built RAG system looks like. If you are weighing this decision and want a second opinion grounded in the specifics of your data, we are happy to have that conversation.