RAG for Everyone

← Back to all chapters

Chapter 1 of 28 Beginner 15 min read

Why RAG Exists

Imagine asking an AI assistant, "Can I report fraud directly to the SEC under our company policy?" A smooth answer is not enough. You need the right answer, from the right document, with a source you can verify.

How to read this chapter

This chapter is not trying to impress you with architecture words. It is trying to give you the first mental switch: a GenAI system becomes more trustworthy when it knows where its evidence came from.
Production lens We will keep the language friendly, but the thinking is real production thinking. If you are new, follow the story first. If you are senior, look for the trade-offs: source trust, access control, retrieval failure, citations, evals, and escalation.
Sub-topic 1 of 5

The Problem RAG Solves

At a glance
  • Large language models are strong at language, but they do not automatically know your latest private documents.
  • A grounded assistant must retrieve approved evidence before writing the answer.
  • The first production question is not "Can the model answer?" It is "Can the system prove where the answer came from?"
Why this matters: most failed GenAI pilots do not fail because the model cannot write. They fail because the product cannot prove accuracy, freshness, access control, or source authority.

In the early days of using large language models, many people treated them like magical encyclopedias. Ask a question, get a confident answer. That felt amazing until the answer sounded right but was quietly wrong, outdated, or completely unrelated to the private documents the user actually cared about.

That is the doorway into RAG. A language model can write fluently, but fluency is not the same as evidence. If an employee asks, "How many vacation days do I get after five years?", the assistant should not guess from general HR knowledge. It should read the company's policy and cite the source.

Plain-language definition

Retrieval-Augmented Generation means: retrieve trusted information first, then generate an answer using that retrieved context.

What is... retrieval? Retrieval means searching a trusted source and bringing back the most relevant pieces before the model writes an answer. In RAG, retrieval is the evidence-gathering step.

A tiny history

First, teams tried prompting models harder. Then they tried pasting longer and longer documents into the chat. Eventually the pattern became clear: serious systems need a retrieval step. Find the right evidence first. Ask the model to answer second.

Go Deeper: why prompting harder is not enough

Prompting can ask the model to be careful, cite sources, or avoid guessing. But if the right source never reaches the model, the prompt has nothing reliable to ground on. In production, good prompting and good retrieval work together: retrieval supplies evidence, and prompting tells the model how to use that evidence safely.

Quick recap
  • RAG exists because fluent answers are not the same as verified answers.
  • Retrieval supplies trusted evidence before generation.
  • Every grounded answer should be traceable back to sources.
Sub-topic 2 of 5

Model Memory vs. Trusted Context

Model memory is what the model learned during training. Trusted context is what your system provides at answer time. RAG is the discipline of choosing, formatting, and testing that context.

Think of the model as a brilliant colleague who writes well, reasons quickly, and sometimes speaks too confidently. RAG is how we hand that colleague the exact binder, page, and paragraph before asking them to respond. 

User question
  -> retrieve policy chunks
  -> build grounded prompt
  -> generate answer with citations
  -> evaluate retrieval and answer quality
Northkeep Policy Assistant RAG architecture diagram showing policy corpus, chunking, embeddings, vector store, retrieval, prompt, answer, and evaluation.
The core RAG move: retrieve trusted evidence before asking the model to write the answer.
Model memory

Good for language and patterns

The model can explain, summarize, reason, and write clearly. But its training knowledge may be outdated, incomplete, or unrelated to your private corpus.

Trusted context

Good for source-grounded truth

Your system provides current approved evidence: policy pages, runbooks, tickets, contracts, product docs, or database records.

Go Deeper: the real contract of a grounded answer

A grounded answer should satisfy three checks: the cited source exists, the cited source actually supports the claim, and the user is allowed to see that source. Missing any one of these turns a polished answer into a product risk.

Sub-topic 3 of 5

The Northkeep Policy Assistant

This repository already has a working RAG engine in 01-policy-rag-poc. It uses synthetic banking HR and compliance policies, local TF-IDF + SVD embeddings, ChromaDB retrieval, Groq or Ollama generation, and a golden retrieval eval set.

We will use that project like our training ground. Not a toy that only works in a demo, and not a giant enterprise system that hides every moving part. It is small enough to understand and real enough to expose the decisions that matter.

Current eval baseline

Recall@3 is 86.7% and MRR is 0.789. Two training-policy misses are preserved as learning material because real systems improve through failure analysis.

Production case study

Why this is not just a demo

Northkeep gives us a controlled production-like surface: private policies, retrieval misses, citation expectations, and measurable evals. That matters because learners can see where RAG breaks instead of only seeing the happy path.

  • Source authority: which policy is the approved source?
  • Retrieval quality: did the system fetch the right policy chunk?
  • Answer grounding: did the final answer stay inside the retrieved evidence?
  • Failure analysis: what do the two misses teach us about chunking, query wording, or source coverage?
northkeep_rag_flow.txt 
question = "Can I report fraud directly to the SEC?"
retrieved_context = retrieve(question)
answer = generate_answer(question, retrieved_context)
evaluate(question, expected_source="10_whistleblower_policy.md")

Text diagram: where the evidence enters

Employee question
  -> retrieve top policy chunks
  -> build trusted context block
  -> prompt model with rules + context
  -> answer with citations
  -> evaluate retrieval and answer quality

This text version matters because architecture should be explainable even without a rendered image. If you can draw it in plain text, you probably understand it.

Sub-topic 4 of 5

GenAI Builder Thought Process

New-age builders do not just ask AI for an answer. They frame the problem, state assumptions, test outputs against evidence, critique failures, and record engineering decisions.

In this chapter, your thought process is simple: when would model memory be risky, and what evidence should the assistant retrieve before it answers?

This is the new learning style we want to model. We are not just asking GenAI to finish tasks. We are learning how to think with it: frame the question, test the answer, challenge the result, and record the decision so someone else can learn from it later.

In the real world
  • Support assistants need product docs, not guesses.
  • Policy assistants need approved internal documents, not generic HR advice.
  • Research assistants need citations, not confident summaries with missing sources.

Production failure mode

The assistant answers from general model knowledge even though the organization has a stricter internal policy.

Engineering checklist

  • Define approved sources.
  • Require citations for policy claims.
  • Log retrieved source ids and answer status.

Metrics to watch

Recall@k, citation precision, unsupported-answer rate, stale-source rate, and escalation correctness.

Sub-topic 5 of 5

Points To Remember

  • RAG is for grounded answers from trusted context.
  • Retrieval is the evidence-gathering step.
  • Good prompts do not rescue bad retrieval.
  • Evaluation turns failures into engineering tasks.
  • The builder owns the judgment, even when AI helps.

Ready to check what stuck? Head to the interactive quiz next. The flow after that is exercises, practice bank, interview questions, and then the project.

← All Chapters Take the Quiz →