RAG Evaluation Playbook: Metrics That Catch Hallucinations

Golden sets beat vibes

Prompt changes feel better until they silently break a support workflow. We maintain golden question sets labelled with expected citations, refusal cases, and acceptable answer rubrics. Without that baseline, every model upgrade is a coin flip.

Golden sets should reflect real traffic distribution, including adversarial and ambiguous queries. Sanitised demo questions create a false sense of quality.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.

Measure retrieval separately from generation

If you only score final answers, you cannot tell whether the model hallucinated or never retrieved the right chunk. We track recall@k, MRR, and hybrid search contribution on labelled retrieval sets.

Generation evals then judge faithfulness to retrieved context, completeness, and tone. Splitting these stages focuses engineering effort on the actual failure mode.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.

Regression gates in CI

Every prompt, chunking, or embedding change runs the eval suite. Soft regressions warn; hard regressions on faithfulness or refusal policy block merge. Human review still matters for ambiguous cases, but CI catches the obvious drops.

We also sample production traces weekly into the golden set so evals track product reality instead of freezing in time.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.

Online signals and feedback loops

Thumbs-down rates, escalation to human agents, and citation click-through complement offline scores. A model that scores well offline but increases escalations is not an improvement.

Close the loop by routing failing online examples into labelled review. Evaluation is a product system, not a one-time notebook.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.

Turn failures into engineering work

Release AI capabilities behind a controllable boundary. Start with internal users or a small tenant cohort, publish an obvious feedback route, and monitor latency, cost, refusal rate, citation coverage, and escalation rate together. A lower cost answer is not a win if it makes agents reopen work later. Set rate limits and monthly budgets before demand grows, preserve a non-AI workflow, and give support a way to report harmful output with context. Production readiness is reached when an on-call engineer can identify what happened, disable the risky path, and return the user to a useful outcome without waiting on the model provider.

Production AI needs a product contract that survives imperfect inputs and changing providers. Define the user decision, the approved data sources, the freshness rule, and the action boundary before tuning a prompt. Keep an example set drawn from real requests, including vague questions, missing context, sensitive data, and attempts to reach another tenant. Review failures by layer: source content, retrieval, instructions, tool execution, or presentation. That diagnosis prevents a cosmetic prompt change from hiding an authorization or data-quality defect. Release gradually, keep a non-AI path available, and make feedback attach to a trace rather than an anonymous thumbs-down. Monitor cost, time to first token, grounded citation rate, refusal quality, and escalation rate together. A feature is operationally ready when the team can explain the answer, reproduce the context, cap spend, disable a risky capability, and still give the customer a useful next step.

For an AI roadmap, make the operating model visible to the whole product team. Product should own the user outcome and refusal experience; engineering should own the data boundary, service reliability, and measurable quality gates; support should own the escalation route; and security should approve the information class before it reaches a provider. Review a small sample of traces on a cadence, including successful answers, expensive answers, and failures. Version prompts, retrieval settings, and tool schemas so an incident can be reproduced rather than discussed from memory. When a source changes, assess retrieval and citations before shipping it. When a provider changes, rerun the same golden tasks. This discipline creates a feedback loop in which AI capability becomes more useful with use instead of becoming an opaque feature that is too risky to improve.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.

Gate every retrieval change

Put faithfulness and recall@k gates in CI for prompt, chunking, and embedding changes. Sample failing production traces into the golden set weekly so evals track reality.

RAG quality is a product system: offline scores without online feedback will drift, and online feedback without labelled review will not teach the team what to fix.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.

Split retrieval scores from answer scores

If you only score final answers, you cannot tell whether the model hallucinated or never retrieved the right chunk. Track recall@k and faithfulness separately so engineering effort hits the real failure mode.

Promote one failing production trace into the golden set every week. Static eval suites freeze quality in the past.