A system becomes dependable when the team can see where it works and where it does not. For traditional software that visibility comes almost free — tests pass or they don’t. For AI systems it must be built deliberately, and most teams build it late.
If you’ve shipped anything with an LLM in the last couple of years, you know the pattern: the demo is magical, the first week in production is fine, and the first time a VIP user sees a confidently wrong answer, everyone suddenly cares about “quality.”
By then, it’s too late to pretend evaluation is a nice-to-have.
This post is about making evaluation a first-class part of your AI product—early, visible, and tied directly to decisions you can actually make.
The Demo-to-Production Gap
Every AI system has two sets of behavior:
The demo behavior — Hand-picked prompts, curated examples, ideal conditions. Everything works. The model sounds like a genius who also happens to be a great writer.
The production behavior — A distribution of real user queries: messy, ambiguous, sometimes adversarial. Mostly fine, occasionally wrong in ways that matter.
Evaluation is the discipline of knowing that distribution before your users do the sampling for you.
In traditional software, you get this almost for free: unit tests either pass or fail, CI tells you when something breaks, and you can reproduce bugs deterministically.
With LLMs, none of that is true by default. The same prompt can yield different answers. The “right” answer is often fuzzy. And the failure modes are subtle: confident hallucinations, missing citations, over-eager answers when the system should have said “I don’t know.”
If you don’t build visibility into this, you’re not shipping a product. You’re shipping a black box with a UI.
Build the Golden Set Early
A golden set is a curated list of inputs with known-good expected behavior. It’s your “unit tests for AI,” but designed for the reality of probabilistic outputs.
Three rules make golden sets actually useful instead of just feel-good artifacts.
1. Mine Real Questions
Synthetic test cases encode the team’s imagination. Production questions encode users’ reality. The gap between those is usually humbling.
Instead of inventing clever prompts:
- Pull real user queries from logs.
- Include edge cases that have already caused issues.
- Add questions that map to your critical workflows (support, compliance, onboarding, etc.).
Your golden set should look uncomfortably like your actual user base.
2. Grade Dimensions Separately
LLM systems don’t fail in one dimension. They fail in many, independently:
- Correctness: Is the answer factually right?
- Citation accuracy: Do the references actually support the claim?
- Abstention behavior: Did the system appropriately say “I’m not sure” when it should have?
- Tone and style: Is it appropriate for your audience?
- Safety and compliance: Does it avoid disallowed content?
A single blended score hides exactly the information you need. You want to know things like “we improved correctness but made the model more overconfident” or “citations got better, but now it abstains too often on easy questions.” Separate scores let you see trade-offs instead of smoothing them away.
3. Version Everything
When behavior changes—model, prompt, retrieval strategy, chunking, embeddings—you need to know whether quality moved, and in which direction.
Without versioned evaluation, you’re guessing. Treat your evaluation setup like code: version the golden set, version the prompts, version the model and retrieval config, and record scores per version.
Then you can say things like: “Switching from Model A to Model B improved correctness by 4% but increased overconfidence on out-of-scope questions by 7%.” That’s not just a metric. That’s a decision you can make.
Abstention Is an Answer
The most underrated design decision in AI products is making “I’m not sure” a first-class output.
There are two failure modes:
Confidently wrong — The system answers from weak or missing context, sounding certain while inventing facts. This destroys trust catastrophically.
Overly cautious — The system refuses to answer anything remotely ambiguous, frustrating users and pushing them back to manual work.
The sweet spot is a calibrated threshold: when the system is confident and has good sources, it answers; when it’s uncertain or lacks grounding, it either abstains or shows its sources and uncertainty explicitly.
Calibrating that threshold is not a technical detail. It’s a product decision with compliance implications, and it deserves explicit sign-off from product, legal, and domain experts—not just engineering.
Questions to answer explicitly:
- In which domains is a wrong answer unacceptable?
- Where is a partial answer acceptable if clearly labeled?
- What does “I’m not sure” look like in your UI?
- How often is it acceptable to see that message before users lose faith?
If you can’t answer these, your evaluation isn’t done.
Close the Loop
Evaluation isn’t a gate before launch. It’s the operating loop after it.
The teams that succeed with AI systems treat them like living products with a changelog, not deployments with a prayer. That means:
- User corrections, overrides, and flags flow somewhere reviewable — not into a void, not into a Slack thread that disappears in a week.
- That review changes something visible — source material, retrieval strategy, abstention thresholds, prompt templates, fine-tuning data.
- You can trace the impact — “We changed X on date Y, and metric Z moved by N%.”
This is how you move from “the model is weird sometimes” to “we know where it’s weird, why, and what we’re doing about it.”
The Goal Is Not a Perfect Score
Perfect scores on synthetic tests are cheap. You can always tune your golden set until the numbers look pretty.
The goal is not a perfect score. The goal is a better next decision:
- For the system: “Given this class of queries, we should answer / abstain / escalate.”
- For the team: “Given this trade-off, we prioritize citation accuracy over coverage in this flow.”
Evaluation is the surface where your AI product becomes legible to the people responsible for it. Without it, you’re flying blind and hoping your users don’t notice. They will.
A Minimal Evaluation Setup You Can Actually Ship
If you’re starting from zero, here’s a pragmatic baseline:
- Collect 50–200 real user queries that matter to your core workflows.
- Define expected behavior for each, including the correct answer (or range), required sources if any, and whether abstention would be acceptable.
- Score each dimension separately — correctness (0–1 or 0–5), citation quality, abstention appropriateness.
- Run this set on every meaningful change — model swaps, prompt rewrites, retrieval changes.
- Track trends over time, not just absolute numbers.
- Wire user feedback (thumbs up/down, corrections, flags) back into this set regularly.
This is not exhaustive. It’s enough to stop guessing and start making informed trade-offs.
The Uncomfortable Truth
Most AI teams don’t lack clever prompts or powerful models. They lack visibility.
Evaluation is the product surface that makes your AI system understandable, correctable, and improvable. It’s where engineering, product, and risk meet.
If you’re not building it early, you’re not building a product. You’re building a demo with a distribution problem.
Make evaluation part of the product surface. Your users—and your future 2am self—will thank you.