Evaluating RAG in production: measure what predicts real outcomes

Most RAG systems fail quietly. Retrieval looks fine in a demo, then real questions arrive and the answers drift. The fix is to evaluate the pipeline the way users actually experience it, and to measure the parts that predict a good answer.

Start by separating retrieval quality from generation quality. If retrieval returns the wrong chunks, no prompt will save the answer. Measure retrieval with context precision and context recall on a labeled set of real questions. Measure generation with faithfulness, so you catch answers that are fluent but unsupported by the retrieved context.

Build the evaluation set from questions your users really ask, not from questions that happen to be easy. Include the messy ones: ambiguous phrasing, questions that span two documents, and questions with no good answer in the corpus. A system that says 'I do not know' when the context is missing is more trustworthy than one that guesses.

Run the evaluation on every change to chunking, embeddings, or the prompt. Small changes move these numbers in ways that are hard to predict, so treat evaluation as a gate, not an afterthought.