Data flow · rag-eval-harness · offline, zero API keys

Offline RAG evaluation harness, measured layer by layer

Offline RAG evaluation harness, measured layer by layer Data flow of an offline, zero-API-key RAG evaluation harness: twelve analytics concept docs are chunked into a TF-IDF index, the twenty labelled questions drive cosine top-5 retrieval and a deterministic extractive answerer, and the retrieval metrics, answer correctness and construction-invariant faithfulness are scored against the same labelled set's gold doc ids and gold answers. 01INDEX 02RETRIEVE 03ANSWER 04SCORE KNOWLEDGEBASE RETRIEVALTF-IDF GENERATIONOFFLINE EVALUATIONSCORING TOP-K CONTEXT GOLD LABELS KB Corpus Docs 12 concept docs data/docs/*.md DB KB Labeled Questions 20 q · queries + gold data/eval.jsonl DB RET TF-IDF Index chunked · 1–2-grams build_index.py DB TB RET Cosine Top-k k=5 · rank-1 is gold retrieve.py TB TB GEN Extractive Answerer verbatim sentence · offline generate.py · zero API key TB FL SCO Retrieval Metrics recall@5 1.00 · MRR 1.00 precision@5 0.20 · k=5 TB FL SCO Faithfulness ≡ 1.00 invariant, not a metric answer copied from context SCO Answer Correctness 0.37 · Jaccard vs gold generation-side signal FL FL NODE PIPELINE STAGE FOCAL · ZERO-KEY ANSWERER INVARIANT · NOT A METRIC DATA DB DATASET TB CHUNKS · TABLE FL ANSWER · METRIC LEFT CHIP = INPUT · RIGHT CHIP = OUTPUT FLOW DATA HANDOFF FOCAL HANDOFF GOLD REFERENCE (PASSIVE)

Solid arrows are the data path: the 12-doc corpus builds the TF-IDF index, the 20 labelled questions drive cosine top-5 retrieval, and the offline extractive answerer returns a verbatim sentence from the retrieved context. Dashed arrows are the passive gold reference — the same data/eval.jsonl items supply both the query set and the gold doc ids / gold answers the scorers compare against. Nothing here reaches the network: the zero-key path is the pipeline, and the Claude branch of generate.py (--llm + key) is a swap point, not a node. Faithfulness sits on a dashed node because it is a construction invariant of the extractive answerer, not a measured score.