The RAG Evaluation Playbook We Actually Use
Golden sets, hybrid retrieval metrics, and regression gates that catch hallucinations before they reach production users.
Golden sets beat vibes
Prompt changes feel better until they silently break a support workflow. We maintain golden question sets labelled with expected citations, refusal cases, and acceptable answer rubrics. Without that baseline, every model upgrade is a coin flip.
Golden sets should reflect real traffic distribution, including adversarial and ambiguous queries. Sanitised demo questions create a false sense of quality.
Measure retrieval separately from generation
If you only score final answers, you cannot tell whether the model hallucinated or never retrieved the right chunk. We track recall@k, MRR, and hybrid search contribution on labelled retrieval sets.
Generation evals then judge faithfulness to retrieved context, completeness, and tone. Splitting these stages focuses engineering effort on the actual failure mode.
Regression gates in CI
Every prompt, chunking, or embedding change runs the eval suite. Soft regressions warn; hard regressions on faithfulness or refusal policy block merge. Human review still matters for ambiguous cases, but CI catches the obvious drops.
We also sample production traces weekly into the golden set so evals track product reality instead of freezing in time.
Online signals and feedback loops
Thumbs-down rates, escalation to human agents, and citation click-through complement offline scores. A model that scores well offline but increases escalations is not an improvement.
Close the loop by routing failing online examples into labelled review. Evaluation is a product system, not a one-time notebook.
Aisha Okafor
AI Engineer
AI engineer focused on RAG evaluation, fine-tuning decisions, and production LLM quality.
