Exercises: Score Answer Quality
Practice turning answer reviews into repeatable rubric scores that can support release decisions.
Your tasks
- Score answer quality across faithfulness, relevance, citation support, completeness, refusal correctness, and safety.
- Detect unsupported terms and bad citations using simple deterministic checks.
- Return a severity and release gate for each case.
- Explain the next engineering action for the team.
Production Scenario
Your team changed the system prompt to make answers more concise. The answers are shorter now, but you need to check whether the model started dropping conditions, citing weak sources, or refusing too often.
Exercise benchmark
supported policy answer -> should pass
unsupported manager claim -> should block
false refusal -> should warn/block depending on risk
wrong citation -> should require review or block