Retrieval Metrics

Why Retrieval Metrics Matter

Imagine a user asks: "What approval is needed for enterprise refunds after 30 days?" The answer depends on a very specific policy chunk. If retrieval misses that chunk, the model has only bad options: guess, refuse, or answer from a nearby but wrong policy.

Retrieval metrics isolate this first critical question: did the system retrieve the evidence that could support a correct answer? This is different from asking whether the final answer sounded good.

Real-Life Analogy: Search Party

Think of retrieval as a search party looking for a critical file in a warehouse. Recall asks, "Did they find the file at all?" MRR asks, "How quickly did they find it?" Precision asks, "How much irrelevant material did they bring back?" nDCG asks, "Did the best and most authoritative evidence appear before weaker evidence?"

This analogy matters because production RAG has the same problem. If the right file is found at position 40, the search party technically found it, but the person answering the question may only inspect the first 5 files. Rank and cutoff matter.

Gold Evidence

Gold evidence is the expected evidence for a query. It is the answer key for retrieval evaluation. Without gold evidence, teams often argue based on vibes: "These results look okay to me." That does not scale.

query_id: q_refund_001
question: "Who approves enterprise refunds after 30 days?"
gold:
  - chunk_id: policy-v4:refunds:enterprise-exception
    relevance: 3
    reason: "contains VP approval rule"
  - chunk_id: addendum-v2:enterprise-refunds
    relevance: 2
    reason: "contains written reason requirement"
slice:
  domain: policy
  risk: high
  query_type: exact_policy_lookup

Notice the relevance grade. This lets us evaluate not only whether any source appeared, but whether the best source appeared early.

Recall@K

Recall@K asks: "Out of all gold evidence items, how many appeared in the top K retrieved results?" It is usually the first metric to build because evidence that is never retrieved cannot be used by the model.

gold chunks: S2, S9
retrieved top 5: S1, S2, S4, S8, S9
hits: S2, S9
Recall@5 = 2 / 2 = 1.0

Now compare a dangerous case:

gold chunks: S2
retrieved top 50: S1, S3, S4, ... S2 at rank 42
Recall@50 = 1.0
Recall@5 = 0.0
final context uses top 5
product outcome: answer fails

Precision@K

Precision@K asks: "Of the top K results, how many are relevant?" Precision matters because the prompt has limited space. Noisy context can distract the model, crowd out better evidence, and increase cost.

Mean Reciprocal Rank

MRR rewards systems that put the first relevant result near the top. It is especially useful when one good source is enough to answer the question.

query A first relevant rank = 1 -> 1/1 = 1.0
query B first relevant rank = 4 -> 1/4 = 0.25
query C first relevant rank = not found -> 0.0
MRR = average reciprocal rank across queries

If Recall@10 is stable but MRR drops, the system still retrieves gold evidence, but pushes it lower. This usually means a ranking, fusion, or reranking regression.

nDCG

nDCG helps when relevance is graded. Some evidence is perfect, some is partially useful, and some is weak background. nDCG rewards putting stronger evidence earlier.

3 = directly answers the question
2 = supports one condition or exception
1 = related background
0 = irrelevant

Question: "What changed in refund policy v4?"
Best ranking: policy-v4 changelog (3), refund policy v4 (3), enterprise addendum (2), old policy (1)

Use nDCG when your system must compare multiple useful sources, not just find one correct source. Legal, research, policy, and incident runbook systems often benefit from graded relevance.

Hit Rate

Hit rate asks whether at least one relevant item was retrieved. It is simple and useful for health dashboards, but it hides rank and completeness.

Failure Slices

Average metrics can lie. A system may show 86% Recall@10 overall while failing 40% of exact policy-id queries. Production teams slice metrics because users do not experience averages; they experience their specific workflow.

overall Recall@10: 0.86
policy Recall@10: 0.91
support Recall@10: 0.88
exact_terms Recall@10: 0.52  <-- action needed
pdf_tables Recall@10: 0.43   <-- parsing/indexing issue likely

Architecture View

gold query set
  -> run retriever against candidate index
  -> capture retrieved ids, ranks, scores, filters, route
  -> compute Recall@K, Precision@K, MRR, hit rate, nDCG
  -> slice by domain/query/source/risk
  -> compare against baseline
  -> release, warn, or block

Dashboards

A production dashboard should show trends, not just one score. Track metrics by index version, retriever route, embedding model, reranker version, prompt version, and source freshness.

A strong dashboard answers: what changed, where it changed, who is affected, and what engineering action should happen next.

Release Gates

Retrieval changes should pass gates before rollout. A new embedding model, chunking strategy, metadata filter, hybrid route, or reranker can improve one slice and break another.

Common Mistakes

Interview-Level Understanding

A strong answer sounds like this: "I evaluate retrieval separately from generation. I label gold evidence, run benchmark queries, compute Recall@K, Precision@K, MRR, hit rate, and sometimes nDCG, then slice by domain and query type. If recall is low, I inspect corpus, parsing, chunking, filters, query rewriting, and retrieval strategy. If recall is high but MRR is low, I inspect ranking, fusion, and reranking. I use those metrics as release gates, not just dashboards."