RAG Evaluation Harness — retrieval and generation metrics
▶ Data-flow diagram — self-contained HTML, opens in a browser — no install, no build
Situation
A RAG pipeline is easy to demo on one lucky question and almost impossible to measure: you cannot tell where quality is lost — in document retrieval or in answer generation — or whether the result reproduces at all without external services. Without an offline harness, any “it got better” stays an impression, not a number.
Task
I needed a RAG system built end-to-end, with each layer measured separately: retrieval metrics (recall@k, MRR, precision@k) over a labeled question set, and generation metrics (faithfulness, answer correctness). The harness has to run with zero API keys and stay unchanged when an LLM is plugged in.
Actions
The corpus is 12 short analytics concept docs (A/B testing, CUPED, uplift, retention, churn, RFM, funnel, LTV, attribution, cohort, DAU/MAU, SQL window functions), chunked and TF-IDF-indexed with 1–2-grams.
- Retrieval — cosine top-k over the TF-IDF matrix; dependency-light and fully offline. An embeddings backend can be dropped in without touching the evaluation code.
- Generation — a Claude model grounded in the retrieved context if a key is set; otherwise a deterministic extractive fallback returns the most relevant sentence from the top doc. Both paths return the same
Generationrecord, so evaluation is reproducible offline. - Evaluation — 20 labeled questions (
data/eval.jsonl) with gold doc ids and gold answers. Retrieval: recall@k, MRR, precision@k. Generation: faithfulness (share of answer tokens grounded in the retrieved context) and answer correctness (Jaccard overlap with the gold answer).
| Metric | Value |
|---|---|
| recall@5 | 1.00 |
| MRR | 1.00 |
| precision@5 | 0.20 |
| faithfulness (invariant, extractive) | 1.00 |
| correctness | 0.37 |
Run
uv run --with scikit-learn --with pandas --with numpy python run.py query "How is the CUPED theta coefficient computed?"
uv run --with scikit-learn --with pandas --with numpy python run.py eval
# with a real LLM (requires ANTHROPIC_API_KEY):
uv run --with scikit-learn --with pandas --with numpy --with anthropic python run.py eval --llm
uv run --with scikit-learn --with pandas --with numpy --with pytest pytest -qPer-question breakdown: reports/retrieval_per_question.csv, reports/generation_per_question.csv.
Result
Retrieval is perfect on this corpus: recall@5 and MRR = 1.00, the gold doc ranks 1 for all 20 questions — TF-IDF is enough at this scale and topic separation. The offline extractive baseline is grounded by construction — faithfulness ≡1.00 because the answer is taken from the context — and reaches 0.37 correctness by token overlap.
The value is not the numbers themselves but that the harness separates the layers. With --llm and an API key, generation scores rise while retrieval metrics stay put, so you can see which layer to improve. It is the same principle as an A/B test: measure first, change second.
Limitations
Faithfulness here is a grounding proxy (answer tokens in the retrieved context), not a truth check: an answer can be grounded and still be wrong. Correctness by Jaccard is sensitive to wording — a low score is not always an error. The corpus is small (12 docs, 20 questions), so recall@5 = 1.00 is a property of this set, not a promise on real data; grow the corpus and the metric has to be recomputed.