A language model can produce a convincing answer from a bad context window. That is what makes retrieval systems feel magical in demos and unreliable in production. The gap between those two states is rarely about the model. It’s about everything upstream of it.
If you’ve built or shipped a RAG (Retrieval-Augmented Generation) system, you know the pattern: the demo is stunning — the model answers complex questions, cites sources, and sounds like an expert who actually works at your company. Two weeks into production, someone asks a slightly different question and gets a confident, well-written, completely wrong answer. Suddenly, “quality” becomes everyone’s top priority.
By then, most teams realize the hard truth: the model was never the bottleneck.
This post is about what RAG actually needs to work in production—and why most of it lives far upstream of the LLM.
Retrieval Is a Product Surface
The most common mistake we see in RAG projects is treating retrieval as plumbing—an implementation detail that happens before the “real” work of prompting and generation.
In practice, retrieval is the product.
The user’s question succeeds or fails at the moment context is selected. If the right documents aren’t in the context window, no amount of prompt engineering will save you. The model will just hallucinate more elegantly.
That reframing changes what you build.
Vocabulary First
The words people actually use to ask questions rarely match the words documents use. Users ask “How do I reset my password?” while docs say “Password recovery procedure for user accounts.” Users ask “Can I expense this?” while docs say “Reimbursable business expenses policy.”
If your retrieval system doesn’t bridge that vocabulary gap, it doesn’t matter how good your embeddings are.
Before choosing any chunking strategy or tuning any hyperparameters: collect real user questions, map them against the language in your corpus, and identify mismatches to build synonyms, aliases, and query expansions around those gaps.
This is not “search optimization.” This is product design.
Chunking Follows Structure, Not Token Counts
A policy document’s meaning lives in its sections, headings, and logical units—not in arbitrary 512-token windows.
Splitting on token boundaries cuts off context mid-thought, separates claims from their supporting conditions, and destroys exactly the structure that made an answer trustworthy.
Better approach: chunk by document structure (sections, subsections, logical blocks), preserve headings and metadata in each chunk, and overlap chunks at boundaries to avoid losing context.
Your goal isn’t to maximize token efficiency. It’s to preserve semantic completeness.
Hybrid Beats Pure
Keyword search still outperforms embeddings for identifiers (ticket numbers, policy codes, employee IDs), exact names (product names, internal tools, acronyms), and codes and references (section numbers, clause IDs).
Vector search handles paraphrase and conceptual similarity.
Production systems almost always need both, plus a fusion layer tuned against real questions:
- Run keyword search (BM25 or equivalent).
- Run vector search.
- Fuse results with learned or heuristic weights.
- Tune those weights against your golden set.
Pure vector RAG is a great demo. Hybrid retrieval is what survives production.
Evaluation Before Generation
You cannot improve what you cannot see.
The highest-leverage artifact in any RAG engagement is not the pipeline. It’s a golden set of questions with known-good answers, built from actual user questions rather than synthetic ones.
Build the Golden Set from Reality
Synthetic questions encode the team’s imagination. Production questions encode users’ reality. The gap between those is usually humbling.
To build a useful golden set: mine real queries from logs, support tickets, and chat transcripts; include edge cases that have already caused issues; and cover your critical workflows.
For each question, define the correct answer (or acceptable range), the required sources, and whether abstention would be appropriate. This becomes your ground truth for every change you make.
Grade Three Things Separately
RAG systems don’t fail in one dimension. They fail in many, independently:
- Correctness of the answer — Is the information factually right and complete?
- Accuracy of citations — Do the referenced sources actually support the claim? Are they the right documents?
- Abstention behavior — Did the system appropriately say “I’m not sure” when it should have?
A single blended score hides exactly the information you need. Separate scores let you make informed decisions instead of chasing a vague “quality” number.
Calibrated Honesty Is a Feature
A system that answers confidently from weak context scores worse than one that says “I’m not sure, but here are the closest sources.”
That third behavior—calibrated honesty—is what separates systems people trust from systems they quietly stop using.
This isn’t just a technical setting. It’s a product decision with compliance implications, and it deserves explicit sign-off from product, legal, and domain experts.
Context Windows Are Not Knowledge
The model doesn’t know your business. It knows how to read.
Everything it says about your domain comes from what you retrieve. This has uncomfortable implications teams prefer to defer.
Stale Sources Produce Confident Nonsense
If your retrieval system pulls from outdated docs — policies that changed last month, processes deprecated in Q1, product features that no longer exist — the model will answer confidently, using stale facts.
Freshness metadata belongs in retrieval scoring: prefer newer documents when content is similar, deprecate or flag outdated sources, and build expiration or review dates into your corpus.
A RAG system is only as good as its underlying knowledge—and that knowledge must be maintained.
Permissions Must Flow Through Retrieval
An answer the user wasn’t allowed to read is a security incident with good grammar.
Your retrieval layer must respect document-level permissions, role-based access, and data classification (internal, confidential, public). If your vector store doesn’t support filtering by permissions, you don’t have a production system. You have a demo with a compliance problem.
Contradictory Sources Need Adjudication Rules
Real corpora contradict themselves: old policy vs. new policy, regional variations, exceptions for specific teams or customers.
Averaging over contradictions doesn’t work. You need explicit rules — prefer newer over older, prefer official policy over team wikis, prefer global over local unless overridden. Encode these rules in retrieval scoring or post-retrieval ranking. Don’t leave it to the model to “figure out.”
The Unglamorous Path Works
The best production RAG systems are less theatrical than their demos. They cite sources inline, refuse gracefully (“I’m not sure” is a first-class output, not a failure mode), and log every answer with enough context for a human to audit later.
And critically: feedback flows back into source material, not into opaque fine-tuning. User corrections, flags, and overrides should change the corpus (fix wrong docs, add missing ones), the retrieval (adjust scoring, add synonyms), and the abstention thresholds.
That feedback loop is the actual engine of improvement.
A Practical Checklist for Production RAG
Retrieval
- Map real user vocabulary against corpus language.
- Chunk by document structure, not token counts.
- Use hybrid retrieval (keyword + vector) with a fusion layer.
- Filter by permissions and data classification.
- Score by freshness and source authority.
Evaluation
- Build a golden set from real user questions.
- Grade correctness, citation accuracy, and abstention separately.
- Version your golden set, prompts, and retrieval config.
- Run evaluation on every meaningful change.
Product & Operations
- Make “I’m not sure” a first-class output.
- Show sources inline, not buried in a footer.
- Log answers with full context for auditability.
- Wire user feedback back into corpus and retrieval.
This isn’t exhaustive. It’s enough to stop guessing and start improving systematically.
The Uncomfortable Truth
Most RAG failures aren’t model failures. They’re retrieval, evaluation, and product failures.
The model is the easiest part to swap. The hard parts are the questions your users actually ask, the corpus you maintain, the evaluation harness you build, and the honesty of your interface.
Pick a competent model. Then spend your ambition where the leverage is: upstream of generation.
That’s where production lives.