Faithfulness vs relevance?
Faithfulness checks whether the answer is supported by evidence. Relevance checks whether it answers the user's actual task. An answer can be faithful but still too broad, incomplete, or not useful.
Interview Questions: Answer Quality Metrics
These answers are written for production RAG interviews, design reviews, and architecture conversations.
Faithfulness vs groundedness?
Faithfulness asks whether the claim is supported. Groundedness asks whether we can trace the claim back to a specific source, chunk, citation, version, and authority path.
Why citation correctness?
A citation is useful only when it supports the attached claim. Citation presence alone can create false trust.
How do you evaluate refusal quality?
Track both missed refusals and false refusals. Missed refusals create safety risk. False refusals make the product unusable when evidence is actually sufficient.
What should an answer-quality rubric include?
Faithfulness, relevance, citation support, completeness, refusal correctness, safety, tone, severity, and release decision.
Why is completeness different from relevance?
Relevance checks task fit. Completeness checks whether required conditions, exceptions, steps, or caveats are present.
How do you prevent reviewer drift?
Use rubric examples, calibration rounds, disagreement review, and clear pass/warn/block definitions.
How do you connect answer metrics to retrieval metrics?
Evaluate retrieval first to know whether evidence was available. Then evaluate answer quality to see whether the model used that evidence correctly.
What is an incident replay set?
A benchmark of historical failures that must keep passing after prompt, model, retriever, or index changes.
How should release gates work?
Gate by risk and slice. A single high-risk contradicted claim can block release even if average scores look healthy.
What should be stored in a score trace?
Question, answer, evidence ids, citations, rubric scores, failed checks, prompt version, model version, retriever version, corpus version, risk, reviewer, and decision.
Give a senior-level answer.
I would evaluate answer quality with a rubric that separates faithfulness, relevance, citation support, completeness, refusal correctness, and safety. I would slice by risk and workflow, calibrate human reviewers, keep incident replay sets, and use severity-based release gates.