Answer Quality Metrics

Why Answer Quality Metrics Matter

A RAG pipeline can retrieve the correct source and still fail. The answer may overstate the policy, miss an exception, cite a source that does not support the sentence, refuse unnecessarily, or answer a neighboring question instead of the real one.

This is why answer quality evaluation must sit after retrieval and before release. It treats the final answer as a product artifact, not as a nice paragraph.

Real-Life Analogy: Judge and Evidence

Imagine a judge hearing a case. A confident speech is not enough. The judge asks: what evidence supports this claim? Is the evidence relevant? Is the cited document the correct version? Did the argument answer the actual question?

Answer evaluation works the same way. The answer is the argument. Retrieved context is the evidence. Metrics are the courtroom checklist.

User question
  -> answer claims
  -> cited evidence
  -> support check
  -> usefulness check
  -> safety/refusal check
  -> pass, warn, block, or human review

Faithfulness

Faithfulness asks whether the answer stays inside the evidence. If the evidence says "VP approval is required after 30 days" and the answer says "manager approval is enough," faithfulness failed.

Evidence: "Enterprise refunds after 30 days require VP approval."
Good answer: "Enterprise refunds after 30 days require VP approval."
Bad answer:  "Managers can approve enterprise refunds after 30 days."
Failure: contradicted claim

Groundedness

Groundedness checks whether the answer can be traced back to retrieved context. Faithfulness asks, "Is it supported?" Groundedness asks, "Can we point to where it is supported?"

answer claim
  -> citation id
  -> source chunk
  -> source document
  -> source version
  -> authority metadata
  -> access/freshness status

This matters when the same topic appears in many documents. A policy summary may be related, but the current versioned policy clause may be the only authoritative source.

Answer Relevance

An answer can be faithful but still not helpful. If the user asks for refund exceptions and the answer gives a general refund-policy summary, relevance is weak. Relevance measures whether the answer resolves the user's actual task.

Citation Correctness

Citation correctness is deeper than citation presence. The cited source must directly support the sentence. This is one of the biggest gaps in shallow RAG demos.

Answer: "VP approval is required after 30 days [FAQ-1]."
FAQ-1: "Refunds are usually reviewed by the support team."
Actual source: "Enterprise refunds after 30 days require VP approval."
Problem: citation exists, but it does not support the claim.

Refusal Correctness

RAG systems can fail by answering when they should refuse, or refusing when they should answer. Track both missed refusals and false refusals. Both hurt trust in different ways.

Refusal quality is especially important for legal, finance, health, security, HR, and production operations workflows.

Architecture View

Once the individual dimensions are defined, they need to run as a pipeline: a golden question, its evidence, and the candidate answer go into an evaluator that scores each dimension separately, rolls those scores into an aggregate report, and hands the result to a release gate.

golden question + evidence + candidate answer
  -> quality evaluator (faithfulness, groundedness, relevance, citation, refusal)
  -> per-dimension scores
  -> aggregate report
  -> slice by risk/domain
  -> compare with baseline
  -> pass, warn, block, or human review

Rubric Design

A useful rubric has clear fields and examples. Without examples, reviewers drift. One reviewer gives a 5 because the answer sounds nice; another gives a 2 because a citation is weak. The rubric must reduce that drift.

faithfulness: 0 = contradicted, 1 = partially supported, 2 = fully supported
relevance: 0 = misses task, 1 = partial, 2 = solves task
citation_support: 0 = missing/wrong, 1 = related, 2 = exact support
completeness: 0 = misses required conditions, 1 = partial, 2 = complete
refusal_correctness: 0 = wrong refusal behavior, 1 = imperfect, 2 = correct
safety: 0 = unsafe, 1 = needs review, 2 = safe

Human Review

Use human review for high-risk and ambiguous cases: legal, finance, medical, security, executive policy, HR, production incidents, and customer-impacting answers. Human review should calibrate automated metrics, not replace all automation.

Release Gates

A prompt, model, retriever, or index release should be blocked if it introduces high-risk unsupported claims, citation-support regressions, unsafe refusals, or known incident replays.

candidate prompt/model
  -> run benchmark answers
  -> score rubric dimensions
  -> compare with baseline
  -> slice by risk/domain/query type
  -> pass, warn, block, or human review

Common Mistakes

Interview-Level Understanding

A strong answer sounds like this: "I evaluate RAG answers on separate dimensions instead of one vague quality score: faithfulness against the retrieved evidence, groundedness so every claim can be traced to a specific source and version, answer relevance to the user's actual task, citation correctness so a cited source actually supports the exact sentence it is attached to, and refusal correctness so I track missed refusals and false refusals separately. I score these with a rubric that has explicit examples so reviewers don't drift, and I route high-risk or ambiguous cases to calibrated human review. Above all, I gate releases by risk slice rather than by average score, so one contradicted claim in a legal, finance, or health answer can block a release even when the overall numbers look fine."

Project Roadmap

Build an answer quality scorer that receives question, evidence, answer, citations, expected behavior, and risk. It should return faithfulness, relevance, citation correctness, completeness, refusal correctness, severity, and release gate.