AI engineering · ai · rag · evaluation
Evaluating retrieval-augmented generation
A useful RAG evaluation separates retrieval quality from answer quality and tests both against real questions.
By Azhar · When a retrieval-augmented generation system gives a poor answer, the cause may be missing evidence, weak ranking, or a model that fails to use the evidence it received. Looking only at the final response makes those problems hard to distinguish.
Build a representative question set
Collect real questions users ask, including ambiguous questions, questions with no answer in the corpus, and questions whose answer depends on a particular source. Keep the expected evidence and a short reference answer for each one.
Score retrieval and generation separately
For retrieval, measure whether the needed passages appear near the top of the results. For generation, check whether the answer is supported by those passages, answers the question, and abstains when evidence is absent. Human review remains useful for judging whether a response is genuinely helpful.
Evaluate changes as a system
Compare chunking, metadata filters, embedding models, rerankers, prompts, and model versions against the same evaluation set. Track latency and cost alongside quality. A small improvement in answer quality may not justify a large increase in latency for every use case.
The goal is not one universal RAG score. It is a repeatable way to see which stage improved, which regressed, and whether the change matters to the people using the system.