Practice Bank: The RAG Query Lifecycle
Each scenario asks you to map the lifecycle like a production engineer: route the request, retrieve the right evidence, protect access, generate safely, and log enough to debug.
Product Scenarios
HR Leave Policy
An employee asks about parental leave after moving from the US to Germany.
Customer Refund Bot
Refund eligibility depends on product tier, country, purchase date, and policy version.
Contract Assistant
A PM asks whether a customer contract allows model-output logging.
Sales Enablement
A seller asks if they can promise a restricted enterprise discount.
Healthcare Demo
A clinician asks whether real patient data can be used in a product demo.
Finance Compliance
An analyst asks whether account identifiers can be included in a report.
Platform Scenarios
Security Runbook
An engineer asks what to do when a production API key is leaked.
Developer Docs
A developer asks why a deployment failed after a new environment variable.
Failure Cases
Stale Policy
The assistant answers from last quarter's policy after the index missed a new upload.
Wrong Reranking
Hybrid search retrieves the right document, but reranking pushes it below the context cutoff.
Access-Control Leak
Retrieved context includes a confidential clause the user should not see.
Multi-Hop Question
The answer needs both the incident runbook and customer communication template.
1 Production judgment task
Pick four scenarios. For each, write lifecycle map, logs, metric, refusal rule, and first engineering experiment.