Project: Answer Quality Scorer
Build a scorer that grades generated answers before release. This becomes the answer-evaluation brain for prompt, model, retriever, and corpus changes.
Production roadmap
- Define a rubric: faithfulness, relevance, citation support, refusal correctness, completeness, and safety.
- Implement deterministic checks first: citation ids, expected terms, unsupported terms, refusal behavior, and source support.
- Produce severity-based gates: pass, review, or block.
- Summarize failures by domain, risk, and query type.
- Store score traces by prompt, model, retriever, corpus, and benchmark version.
Product Standard
This project is intentionally not a toy "score equals three booleans" script. It models the first version of an evaluator a production team could extend with LLM-as-judge, human review, and observability.
Rubric
Explain every score
Each answer returns passed checks, failed checks, severity, and the first action.
Risk
Gate by severity
High-risk unsupported or missed-refusal failures block release.
Trace
Debug releases
Output includes versions so teams know what changed.