Metric report
Return Recall@K, Precision@K, reciprocal rank, hit rate, and first relevant rank per case.
Exercises: Calculate Retrieval Metrics Like a Release Engineer
In this exercise, you are not calculating metrics for decoration. You are building the small evaluator that decides whether a retrieval change is safe enough to ship.
Your production task
- Compare retrieved ids against gold evidence for several realistic query types.
- Compute Recall@K, Precision@K, MRR, first relevant rank, and hit rate.
- Separate candidate discovery failures from ranking failures.
- Flag filtered gold and stale evidence as their own engineering risks.
Scenario
Your team wants to change the retrieval stack from vector-only search to hybrid search with reranking. Before the change goes live, you need a benchmark report that explains whether the new stack is safer, noisier, or risky for specific slices.
Benchmark cases
policy lookup -> must retrieve the exact policy clause
support error code -> must protect exact terms like ERR_8842
finance table row -> must catch table extraction and stale-document failures
legal access rule -> must explain filtered gold separately from bad searchWhat Good Looks Like
Metric correctness
Numbers are explainable
Each score should be traceable to gold ids, retrieved ids, top-k cutoff, and first relevant rank.
Engineering diagnosis
Failure tells you what to inspect
Low recall points to candidate discovery. High recall with low MRR points to ranking. Filtered gold points to metadata and access rules.
Release readiness
High-risk slices get stricter gates
A finance or legal failure should not be treated the same as a low-risk FAQ miss.
Minimum Acceptance Criteria
Slice summary
Summarize results by domain and query type so averages do not hide failures.
Failure diagnosis
Return a clear next action for missing evidence, buried evidence, stale hits, and filtered gold.
Release signal
Block or warn when a high-risk slice fails its retrieval expectation.