Project: Prompt Regression Harness
Build a test runner that scores prompt answers against golden cases and produces release-gate results. This is the system that stops prompt changes from silently breaking production behavior.
What you will build
- A golden-case test suite with happy path, missing evidence, conflict, and prompt-injection cases.
- Deterministic prompt assertions for citations, refusal, format, and unsupported claims.
- A release-gate summary that blocks high-risk failures.
- A failure report that tells a prompt engineer exactly what to fix next.
Production Milestones
M1: Test Case Schema
Define question, evidence, answer, expected behavior, risk, and prompt version.
M2: Assertions
Implement citation presence, refusal correctness, format stability, and injection resistance checks.
M3: Gate
Block releases on high-risk unsupported answers, missed refusals, and injection failures.
Revisit-Value Extensions
- Add a dashboard that compares prompt versions over time.
- Add reviewer notes for ambiguous failures.
- Add incident replay cases from real user feedback.
- Add model-version comparison to decide whether a prompt or model changed behavior.