RAG for a knowledge base: what survives production

Every company experimenting with generative AI arrives quickly at the same idea: connect the model to the internal knowledge base. The technique has a name, RAG, and the basic architecture is simple. What is not simple is making it survive real data, real permissions and real users.

The model is almost never the problem

RAG was formalised in 2020 by combining two memories: the parametric one held in the model's weights, and a non-parametric one retrieved at question time. The promise is good: you update the index without retraining anything.

When a corporate RAG answers badly, the cause is almost always retrieval, not generation. If the right passage never reached the context, no model saves the answer. That is why the first metric to instrument is not answer quality, it is whether the correct document made it into the top-k.

Chunking is a product decision, not an infrastructure one

Splitting documents into pieces looks like a technical detail and is where most projects lose quality. Too large and the chunk drags noise along; too small and it cuts off the sentence that gave the number its meaning.

The aggravating factor in a company is that an isolated passage usually loses its referent. "The rate becomes 2.5%" does not say for which product, from when, or under which contract. Retrieved on its own, it becomes a wrong answer that looks right.

The mitigation that works is enriching each chunk with context from its source document before generating the embedding. Experiments published by Anthropic show this contextualisation meaningfully reduces retrieval failure, and improves further when combined with lexical search and reranking.

More context is not the fix it looks like

The natural reaction to a bad answer is to raise top-k and push more text at the model. There is a clear limit to that.

The Lost in the Middle study showed that model performance depends on where the relevant information sits inside the context. Accuracy is highest when it appears at the beginning or the end, and drops visibly when it lands in the middle. The curve is U-shaped.

The practical consequence is direct: filling the context window can make the answer worse even when the right passage is in there. Ordering a few correct passages well beats sending many.

Hybrid search and reranking

Vector search alone fails in two cases common inside companies: product codes and exact contract terms. Embeddings capture semantic similarity, not literal equality, so "error 4021" and "error 4012" sit dangerously close together.

The combination that fixes it is hybrid: dense search for meaning, lexical search for exact terms, and a reranker on top to order the final set. It is more infrastructure, and it is the difference between a demo and production.

Measuring without a hand-written answer key

Evaluating RAG looks like it requires a hand-written answer for every question, which does not scale. Frameworks like RAGAS exist to lower that cost, decomposing evaluation into dimensions that can be measured separately, among them whether the answer is actually grounded in the retrieved passages and whether those passages were relevant to the question.

Separating those two dimensions is what gives you a diagnosis. An unfaithful answer with good context is a generation problem. Irrelevant context is a retrieval problem, and no amount of prompt tuning fixes it.

What specifically breaks in a company

Three things, none of them about the model:

• Permissions. The index must not return a passage from a document the person could not open. Permission filtering has to happen during retrieval, not afterwards on the answer.
• Freshness. A revoked document still sitting in the index produces a confident, wrong answer. Reindexing has to be part of the document lifecycle.
• Provenance. An answer without a link to its source is not auditable and will not earn trust from legal or compliance.

What this means for your team

Start with a small, well-bounded document set that has a clear owner. Instrument retrieval and generation separately from day one. Build a set of thirty to fifty real questions, collected from the people who will actually use it, and use it as a regression ruler for every change to chunking, embeddings or prompts.

And solve permissions in the architecture before you show the demo to leadership. Retrofitting access control into an index that already exists is one of the most expensive refactors in this kind of project.