Chapter 17 of 28Hybrid Retrieval80 min read

Hybrid Search and Query Routing

Hybrid search combines different retrieval signals, usually dense semantic search and sparse keyword search. Query routing decides which retrieval path to use for each question. Together, they help production RAG systems handle both natural language and exact operational facts.

How to read this chapter

Think complementary strengths: dense search understands meaning; sparse search protects exact terms like IDs, error codes, SKUs, and clause numbers.
Do not blend blindly: hybrid search needs fusion logic, score normalization, dedupe, and evals.
Route by intent and risk: a support paraphrase, legal clause lookup, and Sev1 runbook query should not always use the same retrieval path.
Trace every decision: production teams must explain why dense, sparse, hybrid, or fallback was chosen.

Dense retrieval may understand "regenerate credentials." Sparse retrieval may protect "ERR-AZ-4032-B." A production system often needs both.

Core ideaHybrid search combines multiple retrieval methods. Query routing chooses the right method based on intent, domain, exact-term need, risk, cost, and latency.

Sub-topic 1 of 14

Why Hybrid Matters

At a glance

Dense search is strong for meaning and paraphrase.
Sparse search is strong for exact terms and rare tokens.
Hybrid search can improve recall and precision when domains need both.
Routing prevents every query from paying every retrieval cost.

A user asks, "Why did deployment fail with AZ-4032-B after rotating the key?" Dense retrieval may understand key rotation. Sparse retrieval protects the exact error code. Hybrid retrieval can bring both signals together so the answer has the right troubleshooting evidence.

Hybrid retrieval north starUse the retrieval signal that matches the question, and combine signals only when the product needs it.

Hybrid is not automatically better. It is better when dense and sparse solve different parts of the same user need.

Detect exact-term need

Error codes, product IDs, SKUs, legal clauses, and ticket numbers often need sparse retrieval.

Detect semantic need

Paraphrases, vague questions, and natural-language support queries often need dense retrieval.

Choose route

Use dense, sparse, hybrid, graph, or fallback based on intent and risk.

Fuse and evaluate

Merge results, dedupe, rerank, and measure retrieval quality against golden queries.

Sub-topic 2 of 14

Real-Life Analogy: Two Lenses

Imagine reading a document with two lenses. One lens sees the overall meaning. The other lens highlights exact words and codes. If you only use the meaning lens, you may miss a specific policy number. If you only use the exact-word lens, you may miss a paraphrase.

Hybrid search uses both lenses when needed. Query routing decides whether one lens is enough or both should be used.

Mental picture

Dense search: meaning lens.
Sparse search: exact-term lens.
Fusion: merge both views.
Routing: choose the right lens for the question.

Sub-topic 3 of 14

Dense vs Sparse

Dense

Semantic vectors

Good for paraphrase, concept search, natural language, and meaning-based retrieval.

Sparse

Keyword signals

Good for IDs, codes, exact names, rare terms, legal clauses, and product identifiers.

Hybrid

Combined evidence

Good when the user question contains both semantic intent and exact operational facts.

Routing

Choose path

Good when every query should not pay every retrieval cost.

Sub-topic 4 of 14

BM25

BM25 is a classic sparse retrieval method. It scores documents based on query term matches, term rarity, and document length. You do not need to memorize the formula to use the mental model: rare matched terms matter more than common words.

Strong signal

AZ-4032-B, tenant_id, Policy 7.3, SKU-9910, and exact API names.

Weak signal

Vague paraphrases where user words do not appear in the document.

Production role

BM25 is often the sparse side of hybrid search and a useful fallback when dense retrieval misses exact terms.

Sub-topic 5 of 14

Fusion Strategies

Hybrid search must merge results from different retrievers. Dense scores and sparse scores may live on different scales, so simple score addition can be misleading.

Common fusion pattern

Reciprocal Rank Fusion

RRF combines rankings instead of raw scores. If a chunk ranks high in dense search and sparse search, it gets a stronger final rank. This is useful because raw dense and sparse scores are often not directly comparable.

Fusion flow

dense results:  A, C, D, F
sparse results: B, A, E, C

fusion:
  A appears high in both -> strong
  C appears in both      -> useful
  B sparse-only exact hit -> keep if exact term matters
  duplicates             -> collapse before context

Sub-topic 6 of 14

Query Routing

Query routing decides which retrieval path to use. A router can be rule-based, model-based, or hybrid.

Rule-based router

If query contains error code pattern, use sparse or hybrid. If query is natural language, use dense.

Model-based router

Classify intent with a lightweight model or LLM, then choose retrieval path.

Hybrid router

Use rules for obvious exact-term cases and a model for nuanced cases.

Mentor note: routing is a product behavior. Log the route decision so failures can be debugged.

Sub-topic 7 of 14

Risk-Based Routing

Some queries are high-risk: legal, HR, security, finance, medical, compliance, or operational incidents. These may need stricter filters, higher recall, reranking, citations, and refusal behavior.

Routing examples

Simple FAQ: dense search with low k may be enough.
Error code: sparse or hybrid retrieval protects the code.
Policy exception: hybrid plus reranking and parent expansion.
High-risk legal: strict metadata filters, exact clause retrieval, and citation checks.
Sev1 runbook: route to runbook corpus with ordered procedural context.

Sub-topic 8 of 14

Architecture View

Hybrid retrieval routes the query, runs the needed retrievers, fuses results, deduplicates, reranks, and assembles context with citations.

Hybrid trace

{
  "query": "AZ-4032-B after key rotation",
  "route": "hybrid",
  "route_reason": "error code + semantic troubleshooting intent",
  "dense_top": ["key_rotation_runbook:c4", "credential_update:c2"],
  "sparse_top": ["az_4032_b_errors:c1", "key_rotation_runbook:c4"],
  "fusion": "rrf",
  "selected": ["key_rotation_runbook:c4", "az_4032_b_errors:c1"]
}

Sub-topic 9 of 14

Production Examples

E01

Support article paraphrase

Route: dense.

Reason: user words differ from article title, exact IDs are not central.

E02

Error-code troubleshooting

Route: hybrid.

Reason: exact error code plus semantic context both matter.

E03

Legal clause lookup

Route: sparse first, then rerank.

Reason: exact clause and version are critical.

E04

Broad product question

Route: dense or multi-query dense.

Reason: user needs concept coverage across docs.

E05

Sev1 runbook query

Route: domain-specific hybrid.

Reason: runbook domain, exact alert name, and ordered steps matter.

Sub-topic 10 of 14

Metrics

Route quality

Correct route rate, route override rate, route failure label, and high-risk route accuracy.

Retrieval quality

Recall@k by route, MRR, gold rank, exact-term recall, citation support, and duplicate rate.

Operations

Latency by route, search count per query, fusion cost, fallback rate, and empty result rate.

Sub-topic 11 of 14

Common Mistakes

M01

Using dense search for everything

Symptom: exact IDs and error codes get missed.

Better move: route exact-term queries to sparse or hybrid retrieval.

M02

Using sparse search for paraphrases

Symptom: natural language questions miss relevant docs.

Better move: route semantic questions to dense retrieval.

M03

Adding raw dense and sparse scores

Symptom: one score type dominates unfairly.

Better move: normalize or use rank-based fusion such as RRF.

M04

No route trace

Symptom: teams cannot explain why retrieval chose a path.

Better move: log route, reason, retrievers used, fusion method, and selected chunks.

M05

Hybrid for every query

Symptom: cost and latency increase without measurable quality gain.

Better move: route only when hybrid is justified by intent or evals.

M06

Ignoring high-risk routing

Symptom: legal or security answers use weak evidence paths.

Better move: enforce stricter filters, citations, and reranking for high-risk domains.

M07

No exact-term eval slice

Symptom: aggregate recall looks fine while code/SKU queries fail.

Better move: split evals by exact-term, paraphrase, domain, and risk.

M08

Forgetting dedupe after fusion

Symptom: same chunk appears through dense and sparse paths and wastes context.

Better move: dedupe by chunk id, source id, parent id, or content hash.

Sub-topic 12 of 14

Interview-Level Understanding

A strong answer sounds like this: "Hybrid search combines dense semantic retrieval with sparse lexical retrieval. It is useful when queries need both paraphrase understanding and exact-term protection. I route queries by intent and risk, fuse results with calibrated or rank-based methods, dedupe, rerank, and evaluate route-specific Recall@k, exact-term recall, latency, and citation support."

Senior signal

What production reviewers want

Explain when not to use hybrid too. Good architecture avoids unnecessary cost and complexity when dense or sparse alone is enough.

Sub-topic 13 of 14

Mini Assignment

Create a routing policy for five query types: simple FAQ, error code, legal clause, policy exception, and incident runbook. Define retrieval route, filters, fusion method, top-k, reranking need, and metrics.

Sub-topic 14 of 14

Points To Remember

Dense retrieval is strong for meaning; sparse retrieval is strong for exact terms.
Hybrid search combines signals when a query needs both.
Query routing avoids unnecessary cost and chooses the right retrieval path.
Fusion must handle different score scales and duplicates.
Production hybrid retrieval needs route traces and route-specific evals.

Next, test hybrid routing judgment in the quiz, then build a query router in exercises and project.

← Chapter 16 Project Take the Quiz →