A language model can produce a convincing answer from a bad context window. That is what makes retrieval systems feel magical in demos and unreliable in production. The gap between those two states is rarely about the model. It’s about everything upstream of it.

Retrieval is a product surface

The most common mistake we see in RAG projects is treating retrieval as plumbing — an implementation detail that happens before the “real” work of prompting. In practice, retrieval is the product. The user’s question succeeds or fails at the moment context is selected.

That reframing changes what you build:

  • Vocabulary first. The words people actually use to ask questions rarely match the words documents use. We map real queries against corpus language before choosing any chunking strategy.
  • Chunking follows structure, not token counts. A policy document’s meaning lives in its sections; splitting on arbitrary token boundaries destroys exactly the context that made an answer trustworthy.
  • Hybrid beats pure. Keyword search still outperforms embeddings for identifiers, codes, and names. Vector search handles paraphrase. Production systems almost always need both plus a fusion layer tuned against real questions.

Evaluation before generation

You cannot improve what you cannot see. The highest-leverage artifact in any RAG engagement is not the pipeline — it’s a golden set of questions with known-good answers, built from actual user questions rather than synthetic ones.

We grade three things separately: correctness of the answer, accuracy of citations, and abstention behavior. A system that answers confidently from weak context scores worse than one that says “I’m not sure, but here are the closest sources”. That third behavior — calibrated honesty — is what separates systems people trust from systems they quietly stop using.

Context windows are not knowledge

The model doesn’t know your business; it knows how to read. Everything it says about your domain comes from what you retrieve. This has uncomfortable implications teams prefer to defer:

  • Stale sources produce confident nonsense. Freshness metadata belongs in retrieval scoring.
  • Permission structures must flow through retrieval. An answer the user wasn’t allowed to read is a security incident with good grammar.
  • Contradictory sources need adjudication rules, not averaging.

The unglamorous path works

The best production RAG systems are less theatrical than their demos. They cite sources inline. They refuse gracefully. They log every answer with enough context for a human to audit later, and that feedback loop — corrections flowing back into source material, not into opaque fine-tuning — is the actual engine of improvement.

Pick a competent model. Then spend your ambition where the leverage is: the questions, the corpus, the evaluation harness, and the honesty of the interface. That’s where production lives.