Practice Bank: Retrieval Tuning Scenarios
Use starter.py to diagnose retrieval misses and recommend tuning moves.
Scenario 1: Gold chunk at rank 7
Top-k 5 misses evidence.
Scenario 2: Duplicate chunks dominate
Dedupe is needed before context assembly.
Scenario 3: Wrong region policy
Metadata filters are missing.
Scenario 4: Conversational query
Query rewrite can improve retrieval.
Scenario 5: Complex policy exception
Multi-query plus reranking improves recall.
Scenario 6: Low top score
System may need refusal or clarification.
Scenario 7: High k causes noise
Context budget is crowded.
Scenario 8: Filtered-out gold
Metadata quality must be audited.
Scenario 9: Runbook steps out of order
Context assembly must preserve sequence.
Scenario 10: Same score distribution changed
Embedding/index release needs threshold recalibration.