A system becomes dependable when the team can see where it works and where it does not. For traditional software that visibility comes almost free — tests pass or they don’t. For AI systems it must be built deliberately, and most teams build it late.

The demo-to-production gap

Every AI system has two sets of behavior: the one shown in demos and the one distributed across real usage. The second set is a distribution — mostly fine, occasionally wrong in ways that matter. Evaluation is the discipline of knowing that distribution before your users do the sampling for you.

Build the golden set early

A golden set is a curated list of inputs with known-good expected behavior. Three rules make them useful:

  • Mine real questions. Synthetic test cases encode the team’s imagination; production questions encode users’ reality. The gap between those is usually humbling.
  • Grade dimensions separately. Correctness, citation accuracy, and abstention behavior fail independently. A single blended score hides exactly the information you need.
  • Version everything. When behavior changes — model, prompt, retrieval — you need to know whether quality moved. Without versioned evaluation, you’re guessing.

Abstention is an answer

The most underrated design decision in AI products is making “I’m not sure” a first-class output. Systems that answer confidently from weak context lose user trust catastrophically; systems that show their nearest sources when uncertain keep it. Calibrating that threshold is not a technical detail — it’s a product decision with compliance implications, and it deserves explicit sign-off.

Close the loop

Evaluation isn’t a gate before launch; it’s the operating loop after it. User corrections, overrides, and flags should flow somewhere reviewable, and that review should change source material, thresholds, or retrieval — visibly. The teams we’ve seen succeed treat their AI system as a living product with a changelog, not a deployment with a prayer.

The goal is not a perfect score. Perfect scores on synthetic tests are cheap. The goal is a better next decision — for the system, and for the team responsible for it.