Interview Questions: RAG Execution Pipelines

Strong answer: offline prepares trustworthy searchable knowledge; online answers live users with routing, permissions, retrieval, context, answer policy, and tracing.

Red flag: saying RAG is just one vector search plus one model call.

Strong answer: request id, index version, route, filters, source ids, scores, context, prompt version, answer status, latency, cost, and feedback.

Red flag: logging only the final answer.

Strong answer: retry, rewrite, broaden search, ask clarification, refuse, or escalate based on risk and evidence requirements.

Red flag: always answer anyway.

Strong answer: answers depend on the published knowledge version; versioning enables debugging, eval comparison, and rollback.

Red flag: treating indexing as a background script.

Strong answer: before unauthorized documents enter retrieval results or prompt context; UI-only permissions are not enough.

Red flag: trusting the model not to leak retrieved content.

Strong answer: retrieval recall, citation support, route accuracy, refusal accuracy, stale-source rate, p95 latency, and cost thresholds.

Red flag: shipping because three demos looked good.

Strong answer: low-risk queries can retry or answer with caveat; high-risk queries may need refusal or human escalation when evidence is incomplete.

Red flag: one fallback for every request.

Strong answer: defined input/output contract, owner, metrics, failure mode, test coverage, logs, and rollback plan.

Red flag: "it works in the notebook."

Strong answer: inspect retrieved sources first, then filters, route, index version, context assembly, prompt, and final generation.

Red flag: immediately changing the model.

Strong answer: a rule that certain questions need multiple source families, such as access policy plus incident runbook.

Red flag: trusting one nearby source for high-risk answers.