Chapter 16 of 28Retrieval Control80 min read

Retrieval Strategies and Top-K

Retrieval is where RAG decides what evidence the model is allowed to see. Top-k, scores, filters, query rewriting, and multi-query retrieval are not tiny tuning knobs; they shape answer quality, latency, cost, and trust.

How to read this chapter

Think evidence selection: retrieval is not about returning many chunks; it is about returning the right chunks.
Treat top-k as a product decision: too low can miss evidence; too high can flood the prompt with noise.
Use filters before trust: tenant, access, active status, source type, and freshness must shape candidate results.
Debug with traces: scores, ranks, filters, and gold chunk positions tell you what actually happened.

If the right chunk never enters the context window, even the best prompt cannot answer faithfully.

Core ideaRetrieval strategy controls how candidates are found, filtered, ranked, expanded, and passed to the model. Top-k controls how many candidates move forward.

Sub-topic 1 of 14

Why Retrieval Strategy Matters

At a glance

Retrieval quality determines whether the model receives enough evidence.
Top-k changes recall, noise, latency, and token cost.
Scores are ranking signals, not absolute truth.
Filters, rewriting, and multi-query retrieval help when simple similarity search is not enough.

A user asks, "Can an EU annual customer get a refund after renewal?" The answer depends on the right policy chunk, the EU region filter, active document status, and maybe a parent section with exceptions. If retrieval returns generic refund chunks but misses the renewal exception, the answer will be incomplete.

Retrieval north starRetrieve enough relevant, allowed, current evidence for the model to answer without guessing.

Do not maximize top-k blindly. Maximize useful evidence inside the context budget.

Understand query intent

Is the user asking for a definition, policy, comparison, troubleshooting step, or decision?

Apply filters

Restrict by access, tenant, status, region, source type, and effective date.

Retrieve candidates

Use vector, sparse, hybrid, multi-query, or rewritten query strategies.

Assemble context

Deduplicate, expand parent context if needed, cap tokens, and cite exact evidence.

Sub-topic 2 of 14

Real-Life Analogy: A Receptionist Routing Requests

A good receptionist does not send every visitor to every department. They ask enough questions, check permissions, route to the right team, and avoid wasting everyone’s time.

Retrieval does the same job for RAG. The user question arrives. Filters act like permission checks. Search routes to likely evidence. Top-k decides how many people are invited into the room. Context assembly decides who actually gets to speak.

Mental picture

Query: visitor request.
Filters: security desk.
Top-k: shortlist size.
Context: final meeting room.

Sub-topic 3 of 14

Similarity Search

Similarity search compares the query vector with stored vectors and ranks chunks by closeness. It is often the first retrieval strategy, but not the only one.

Good for

Paraphrases, semantic questions, concept matching, support articles, and natural language queries.

Weak for

Exact IDs, rare error codes, stale documents, access rules, and questions needing multiple facts.

Production check

Always inspect whether the gold chunk appears in top-k and whether it passes filters.

Sub-topic 4 of 14

Top-K

Top-k is the number of candidates returned from retrieval. A top-k of 3 returns the three highest-ranked chunks. A top-k of 20 returns more candidates but can add noise and cost.

Too low

Missed evidence

The correct chunk may be rank 6, but top-k 3 never lets it reach the model.

Too high

Context noise

Unrelated or duplicate chunks crowd the prompt and confuse answer generation.

Dynamic

Intent-aware k

Simple FAQ may need k=3; policy comparison may need k=12 plus reranking.

Evaluated

Metric-driven k

Choose k using Recall@k, citation support, latency, and token budget.

Sub-topic 5 of 14

Scores and Thresholds

Scores help rank chunks, but they are not universal confidence values. A score of 0.82 may be strong in one embedding model and weak in another.

Production warning

Score thresholds need calibration

Do not copy a threshold from another project. Build score distributions from your corpus, inspect false positives and false negatives, and tune thresholds with evals.

Threshold thinking

if top_score < calibrated_min_score:
  answer: "I do not have enough evidence"

if gold_chunk_rank > top_k:
  increase k, rewrite query, hybrid search, or rerank

if top chunks are duplicates:
  dedupe before context assembly

Sub-topic 6 of 14

Metadata Filters

Filters narrow retrieval to allowed and relevant chunks. They are not optional in production systems.

Common filters

Access: role, user group, tenant, confidentiality.
Freshness: active status, effective date, expiry date, source version.
Domain: product, region, policy family, source type.
Quality: approved sources, parser quality, document status.

Filters can also cause failures. If metadata is wrong, the gold chunk may be filtered out. Always log filtered-out gold chunks during evals.

Sub-topic 7 of 14

Query Rewriting

Query rewriting transforms the user question into a better retrieval query. It may expand acronyms, remove conversational fluff, add domain terms, or split a complex question.

Useful when

User questions are vague, conversational, multilingual, acronym-heavy, or missing domain terms.

Risk

A bad rewrite can change intent and retrieve confidently wrong evidence.

Control

Log original query, rewritten query, rewrite reason, and retrieval differences.

Sub-topic 8 of 14

Multi-Query Retrieval

Multi-query retrieval searches several reformulations of the same user need. It improves recall when one phrasing misses evidence.

Example

user: "Can I get money back after annual renewal?"
queries:
  1. "annual renewal refund eligibility"
  2. "refund after subscription renewal"
  3. "EU annual plan cancellation refund exception"

merge -> dedupe -> rerank -> context

The trade-off is cost and latency. Multi-query retrieval should be used when recall matters enough to justify extra searches.

Sub-topic 9 of 14

Architecture View

Production retrieval is a pipeline: understand, filter, search, merge, dedupe, rerank, assemble, and trace.

Retrieval trace

{
  "query": "refund after annual renewal",
  "rewrite": "EU annual plan renewal refund exception",
  "filters": {"region": "EU", "status": "active"},
  "top_k": 8,
  "gold_rank": 3,
  "duplicates_removed": 2,
  "context_chunks": ["refund:v3:c12", "refund:v3:c13"]
}

Sub-topic 10 of 14

Production Considerations

Production gear

Dynamic top-k: choose k based on intent, risk, and available token budget.
Dedupe: collapse near-duplicate chunks before context assembly.
Fallback: if dense retrieval is weak, try sparse, hybrid, or ask for clarification.
Thresholds: calibrate score cutoffs on your corpus.
Latency: multi-query and high k increase runtime cost.
Observability: store query, rewrite, filters, top-k, scores, ranks, and citations.
Eval loops: turn misses into golden queries and regression tests.

Sub-topic 11 of 14

Common Mistakes

M01

Using one top-k for every query

Symptom: simple questions get noisy context while complex questions miss evidence.

Better move: use intent-aware or eval-driven k.

M02

Trusting scores as confidence

Symptom: high score chunks are treated as verified truth.

Better move: use scores as ranking signals, then check metadata and citations.

M03

Filtering after context assembly

Symptom: restricted chunks can reach the model.

Better move: enforce filters before retrieval results become prompt context.

M04

No deduplication

Symptom: repeated chunks crowd out diverse evidence.

Better move: dedupe by source, parent, hash, or semantic similarity.

M05

Bad query rewrite changes intent

Symptom: retrieval looks confident but answers the wrong question.

Better move: log original and rewritten query and evaluate rewrites separately.

M06

Ignoring filtered-out gold chunks

Symptom: retrieval seems bad, but metadata filters caused the miss.

Better move: track filtered gold count in evals.

M07

Increasing k to hide poor retrieval

Symptom: recall improves slightly but answer quality drops from noise.

Better move: improve query strategy, hybrid retrieval, chunking, or reranking.

M08

No retrieval traces

Symptom: teams cannot explain why an answer used certain evidence.

Better move: store ranks, scores, filters, rewrites, and selected chunks.

Sub-topic 12 of 14

Interview-Level Understanding

A strong answer sounds like this: "Top-k controls how many candidates retrieval returns, but the right value depends on query type, corpus quality, reranking, context budget, and risk. I tune k with Recall@k, citation support, latency, token cost, and answer quality. I also use filters, query rewriting, dedupe, and hybrid retrieval when simple similarity search misses evidence."

Senior signal

What production reviewers want to hear

Do not say "we set k to 5." Say why, what failures you observed, how k changes by intent, how filters are applied, and how traces prove the gold chunk entered context.

Sub-topic 13 of 14

Mini Assignment

Create a retrieval policy for three query types: simple FAQ, policy exception, and troubleshooting runbook. Define top-k, filters, rewrite behavior, multi-query usage, dedupe, and metrics for each.

Sub-topic 14 of 14

Points To Remember

Retrieval strategy decides what evidence the model sees.
Top-k is a trade-off between recall, noise, latency, and token cost.
Scores need calibration and should not be treated as truth.
Filters, query rewriting, and multi-query retrieval improve control when used carefully.
Production retrieval must be traceable and evaluated with golden queries.

Next, test retrieval judgment in the quiz, then build a top-k policy planner in exercises and project.

← Chapter 15 Project Take the Quiz →