Deployment Patterns

The Deployment Problem

A RAG application has more moving parts than a normal CRUD app: source documents, chunking, embeddings, indexes, prompts, model providers, retrieval parameters, citations, feedback, traces, and eval sets. A release can fail even when the web server is healthy, because the wrong index shipped with the right prompt or because retrieval filters were skipped for one user group.

The deployment question is not "Where can I host this?" It is "What is the smallest architecture that satisfies privacy, reliability, latency, cost, rollback, and support expectations?"

Decision inputs
  audience: public demo | internal users | customers | regulated users
  data: mock | public | private | regulated
  traffic: one user | team | department | internet scale
  risk: learning | workflow helper | business critical | legal/safety critical
  maturity: solo builder | small team | platform team | enterprise ops

Deployment decision framework

1. Classify the data first. Public mock data can be static. Private data needs authentication before retrieval and authorization filters inside retrieval.
2. Classify the promise. A portfolio demo promises learning. A production assistant promises support, uptime, traceability, and rollback.
3. Separate online and offline work. Answering users is latency-sensitive. Crawling, chunking, embedding, and backfills are batch jobs.
4. Version every behavior input. Code, prompt, config, retriever settings, embedding model, index schema, and corpus snapshot must be traceable.
5. Choose the cheapest pattern that still protects trust. A static page is great for mock demos. It is irresponsible for private documents.

Real-Life Analogy: The Fleet-Wide Software Update

An airline that pushes a new flight-management software build never installs it on every aircraft in the fleet the same night. It picks a handful of canary planes on low-risk routes, watches telemetry from those flights closely, and only widens the rollout once the data looks clean. If something looks wrong mid-flight-day, there is a rehearsed way to revert those planes to the previous certified build before the next departure, not a scramble to invent one.

RAG deployments deserve the same discipline. A new embedding model, a rewritten prompt, or a re-tuned reranker is a "software update" for how your system understands and answers questions. Route it to a canary slice of traffic first, watch quality and cost like a flight engineer watches instruments, and keep a rehearsed rollback path ready - covering the code, the prompt, and the index together - so a bad build never gets to fly the whole fleet.

Static Demo Pattern

A static deployment is ideal when the goal is education, portfolio proof, or stakeholder review with mock data. The app can run from HTML, CSS, JavaScript, and precomputed sample responses. There are no server secrets and no live private retrieval.

browser
  -> static assets
  -> sample corpus JSON
  -> precomputed or client-side demo response

No server secrets. No private documents. No write access.

Use it for

• Course projects, documentation demos, sales walkthroughs, and portfolio artifacts.
• Public corpora, synthetic documents, or snapshots that are safe to expose.
• Explaining the user experience before investing in backend operations.

Pros

• Very cheap, fast, cacheable, and easy to share.
• Low operational burden: no runtime, no database, no model key in production.
• Great for visualizing retrieval, citations, and failure states.

Cons

• Cannot protect secrets or private data.
• Cannot enforce real user-specific access control.
• Can mislead stakeholders if sample answers hide real latency, cost, or retrieval failures.

Common mistakes

• Putting API keys in frontend code.
• Using private documents because "the repo is only shared internally."
• Calling a mock demo production-ready because the UI looks polished.

Local CLI Pattern

A CLI is the production engineer's workshop. It makes indexing, evaluation, smoke tests, migrations, and incident reproduction repeatable. Even hosted systems should keep a local CLI path for maintainers.

developer laptop or CI runner
  -> ingest command
  -> chunk/embed command
  -> eval command
  -> smoke query command
  -> report artifact

Use it for

• Rebuilding indexes from a known corpus snapshot.
• Running regression evals before a release.
• Debugging one bad answer with the same retriever parameters used in production.

Trade-offs

The CLI gives reproducibility and low cost, but it is not a multi-user product boundary. It should not become the only place your deployment logic exists. Promote shared behavior into libraries used by both CLI and API.

Concrete commands

rag ingest --source ./docs --snapshot 2026-07-11
rag index --snapshot 2026-07-11 --embedding text-embedding-3-large
rag eval --suite release-gate.yaml --index 2026-07-11
rag smoke --question "How do I reset my badge?" --expect-citation

Backend API Pattern

A backend API is the clean boundary for production RAG. It protects secrets, checks identity, applies authorization filters before retrieval, records traces, enforces rate limits, and gives the frontend a stable contract.

frontend
  -> API gateway
  -> auth middleware
  -> RAG service
       -> policy checks
       -> retriever
       -> reranker
       -> generator
       -> citation validator
       -> trace writer

Design decisions

• Sync or async: use synchronous responses for short answers; use jobs or streaming for long-running workflows.
• Stateless or stateful: keep request handling stateless when possible; store conversations and feedback in explicit databases.
• Single service or split services: start with one service for simpler ownership; split ingestion, retrieval, and generation when scale or team boundaries demand it.
• Provider abstraction: isolate model and vector provider details enough to test fallbacks, but avoid a generic wrapper that hides important limits.

Minimum API contract

POST /answer
  input: user_id, question, conversation_id, filters
  output: answer, citations, confidence_signals, trace_id

GET /health
  checks: app, model_provider, vector_db, trace_store, build_version

POST /feedback
  input: trace_id, rating, correction, unsafe_or_wrong_reason

Hosted Frontend Pattern

The frontend turns the RAG service into a usable product. It must make citations inspectable, uncertainty visible, and recovery paths obvious. Production frontends should not hide source quality behind a chat bubble.

What the frontend owns

• Authentication handoff and session handling.
• Question input, streaming answer display, citation panels, and source previews.
• Feedback capture with trace IDs so operations can reproduce failures.
• Clear error states for rate limits, missing access, provider outages, and empty retrieval.

Architecture choices

• Static frontend plus API: good default for many apps; simple deployment with strong backend controls.
• Server-rendered app: useful when auth, routing, and data fetching are tightly coupled.
• Internal tool shell: best for employee assistants where identity, audit, and permission boundaries matter more than visual polish.

Common mistakes

• Showing generated answers without source snippets or document dates.
• Letting the browser choose retrieval filters.
• Capturing feedback without storing the prompt, retrieved chunks, model version, and index version needed to debug it.

Vector Database Pattern

The vector store is not just a nearest-neighbor box. In production it becomes a governed retrieval system with metadata filters, snapshots, backups, schema migrations, capacity planning, and access policy.

Local index vs managed vector database

Production requirements

• Metadata filters for tenant, document ACL, version, language, freshness, and source type.
• Index versioning so releases can roll forward and back.
• Snapshot and restore tests, not just backup configuration.
• Capacity tests for query concurrency, embedding backfills, and reindex windows.
• Data deletion workflows for retention rules and user requests.

Architecture View

A release candidate is a bundle of code, prompt, and index versions moving together. It earns trust in stages: staging first, then a small canary slice of real traffic, then a monitored ramp that compares quality, latency, and cost against the last good version. Only then does it become the full rollout. If the ramp shows a regression, the rollback gate sends traffic straight back to the last good version while the candidate goes back for a fix - never the other way around.

Operations and Release Controls

Production RAG is operated through evidence. You need to know which version answered, what it retrieved, whether citations supported the answer, how much it cost, and how to return to the last good state.

Release sequence

1. Build code and config with immutable version identifiers.
2. Build or select the index snapshot for the release.
3. Run unit tests, retrieval evals, answer-quality evals, safety checks, and cost estimates.
4. Deploy to staging with production-like auth and data boundaries.
5. Run smoke tests against known questions and expected citations.
6. Canary to a small cohort, compare latency, error rate, retrieval quality, and feedback.
7. Ramp gradually, monitor dashboards, and keep rollback ready until the release is stable.

Production checklist

• Secrets are stored outside code and rotated on a schedule.
• Authentication happens before retrieval; authorization filters are enforced server-side.
• Health checks cover model provider, vector DB, trace store, and index version.
• Every answer records trace ID, prompt version, model version, retriever settings, retrieved chunks, and cost.
• Eval gates block releases that regress groundedness, citation quality, refusal behavior, or latency.
• Rollback covers code, prompt, config, model choice, and index snapshot.
• Rate limits, abuse controls, and budget alerts are active.
• Incident runbooks explain provider outage, bad index, high cost, empty retrieval, and data leakage response.

Common production mistakes

• Deploying the answer API and index rebuild as one fragile process.
• Skipping ACL filters because "the prompt tells the model not to reveal private data."
• Only monitoring uptime while ignoring retrieval miss rate, citation support, and cost per answer.
• Rolling back code while leaving the broken index active.
• Letting evaluation data go stale after the first launch.

Common Mistakes

Interview-Level Understanding

A strong answer sounds like this: "I treat a RAG release as a bundle of versioned code, prompt, and index that moves through staging, a canary slice of real traffic, and a monitored ramp before it becomes the full rollout. Retrieval changes get the same canary discipline as code changes, because a bad embedding model or reranker degrades trust just as fast as a bad deploy. I version the index independently from code so either can roll back without dragging the other along, gate risky retrieval changes behind feature flags, stage multi-tenant rollouts by cohort, and keep a documented, rehearsed rollback runbook so recovery during an incident is a known procedure, not an improvisation."

Project Roadmap

You will build a RAG deployment planner. Given a scenario, it recommends a deployment pattern, explains the trade-off, lists required controls, produces a launch checklist, and names the first day-2 operations tasks.

Milestones

1. Classify the scenario: data sensitivity, traffic, latency SLO, user audience, team maturity, and business risk.
2. Choose the pattern: static demo, local CLI, hosted API, hosted frontend plus API, or enterprise deployment with managed vector DB.
3. Add controls: auth, ACL filters, managed secrets, tracing, eval gates, rollback, rate limits, backups, and feedback.
4. Write the roadmap: demo, staging, canary, production ramp, and operating cadence.
5. Test the planner: run examples for public demos, internal assistants, and regulated customer deployments.

Start with the project brief, then use the exercises and practice bank to pressure-test your decision logic.