Exact IDs have low recall
Likely cause: dense embeddings are smoothing over exact terms. Next move: add sparse or hybrid retrieval and track exact-term hit rate.
Practice Bank: Retrieval Failure Diagnosis
This practice bank teaches the muscle that matters in real teams: when a metric moves, what do you inspect first?
Recall is high, MRR is low
Likely cause: the evidence exists but arrives too late. Next move: inspect fusion, reranking, dedupe, and final context selection.
Gold evidence is filtered out
Likely cause: metadata, tenant, access, or freshness rules are wrong or too strict. Next move: inspect filter traces before tuning embeddings.
PDF table queries fail
Likely cause: parsing and chunking lost table structure. Next move: improve table extraction, row ids, captions, and metadata.
Precision drops after increasing K
Likely cause: candidate recall improved but final context became noisy. Next move: retrieve broadly, rerank tightly, and compress context carefully.
Metrics improve overall but legal slice fails
Likely cause: averages are hiding high-risk regressions. Next move: ship by route or block release until high-risk slices pass.
Drill Format
For each symptom in the code, return four things: likely root cause, first metric to watch, first engineering action, and release decision. This forces the habit of turning a dashboard into a decision.