Project: Retrieval Metrics Dashboard Seed
Build a benchmark runner that computes retrieval metrics and produces release-gate decisions. This becomes the early version of a retrieval quality dashboard.
What you will build
- A benchmark evaluator that compares retrieved ids against gold evidence.
- Recall@K, Precision@K, MRR, and first-relevant-rank calculations.
- Slice-level summaries by domain, query type, and risk.
- A release gate for high-risk retrieval regressions.
Production Milestones
M1: Gold Dataset
Create representative queries with gold chunk ids, source versions, and risk labels.
M2: Metrics Engine
Compute recall, precision, MRR, hit rate, and stale-hit indicators.
M3: Diagnostics
Explain whether failures are recall misses, buried evidence, filter drops, or stale hits.
Revisit-Value Extensions
- Add trend comparison across index versions.
- Add exact-term and semantic-query slices.
- Add charts for candidate recall before rerank and final context precision after rerank.
- Add a release note generator that explains metric changes in plain English.