Practice Bank: Failure Review Scenarios
Use starter.py to score realistic traces and produce release-safe recommendations.
Scenario 1: HR probation exception
Gold evidence exists but sits outside the retrieved child chunk.
Scenario 2: API docs duplicates
Repeated overlap crowds the prompt and lowers context usefulness.
Scenario 3: Runbook warning split
A restart step is retrieved without the safety warning.
Scenario 4: Retired legal clause
Old content wins because active-version filtering is missing.
Scenario 5: Product manual chapter blob
A giant chunk retrieves but citation is too broad to verify.
Scenario 6: FAQ fragments
Tiny chunks retrieve keywords but lose the full answer.
Scenario 7: Mixed policy topics
One chunk contains eligibility, pricing, cancellation, and legal warnings.
Scenario 8: Table boundary break
Rows are split away from column headers and units.
Scenario 9: Code block without explanation
Snippet retrieves, but setup constraints are outside context.
Scenario 10: Correct chunk ranked too low
The chunking is acceptable, but retrieval strategy needs work.