Practice Bank: Reranking Scenarios
Use starter.py to decide whether reranking, retrieval tuning, or filtering is the right fix.
Scenario 1: Gold at rank 18
Reranking can help if candidate k includes it.
Scenario 2: Gold missing entirely
Fix first-stage retrieval.
Scenario 3: Duplicate parent sections
Dedupe and diversify.
Scenario 4: Reranker timeout
Fallback to first-stage ranking.
Scenario 5: High-risk legal answer
Rerank and citation-check.
Scenario 6: Low-risk FAQ
Skip reranking if baseline passes.
Scenario 7: Candidate pool too large
Latency rises.
Scenario 8: Stale candidate promoted
Filters must run first.
Scenario 9: Hybrid result mix
Reranking can choose best evidence.
Scenario 10: Regression after model change
Run before/after evals.