If you’ve built a RAG demo, you know how good it feels. You load a handful of PDFs, chunk them, embed them, drop them into a vector database, wire up a chatbot, and within an afternoon you have something that looks like magic. Then you put it in front of real users with real documents, and it falls apart.

Answers are vague. Citations point to the wrong section. The chatbot confidently quotes a policy that was superseded two versions ago. Retrieval returns five nearly-identical chunks and misses the one paragraph that actually answers the question. None of this is a model problem. It’s an infrastructure problem.

This post is about what changes when you move RAG from a demo to production: the pipeline gets a lot bigger, and most of the engineering effort moves away from the LLM and into the data layer around it.

Why Simple RAG Works for Demos but Fails in Production

The classic RAG pattern is simple enough to explain in one sentence:

documents → chunks → embeddings → vector database → chatbot

It works well when:

  • You control the source documents and their format.
  • There’s no access control to worry about.
  • Freshness doesn’t matter — the corpus barely changes.
  • “Roughly relevant” is an acceptable answer.

Production systems violate every one of these assumptions. Documents come from a dozen different sources in a dozen different formats. Different users are allowed to see different things. Content changes weekly. And “roughly relevant” isn’t good enough when someone is asking about a compliance policy or an incident runbook.

The Old RAG Pattern

To be fair to the simple pattern, it’s not wrong — it’s incomplete. It handles exactly one concern: turning text into vectors you can search semantically. It has no opinion about:

  • Where the data came from or whether it’s still valid.
  • Who is allowed to see it.
  • Whether a chunk boundary cut a sentence in half.
  • Whether a keyword match would have been better than a semantic one.
  • Whether the retrieved context actually supports the generated answer.

Once you start caring about those questions, you’re no longer building a chatbot feature. You’re building a document intelligence pipeline, and the LLM is just the last step in it.

The Modern Document Intelligence Pipeline

In production, RAG looks more like this:

Sources
  → extraction / parsing
  → cleaning and normalization
  → chunking
  → metadata enrichment
  → vector + keyword indexing
  → hybrid retrieval
  → reranking
  → context assembly
  → LLM answer with citations
  → evaluation and observability

Every stage above is a place where quality is won or lost. Let’s go through them.

Extraction and Parsing

Real organizations don’t keep their knowledge in one clean format. You’ll be ingesting:

  • PDFs — often scanned, with tables, multi-column layouts, and headers/footers that need to be stripped.
  • Word documents — with tracked changes, comments, and embedded images.
  • Cloud storage (Drive, SharePoint, S3) — where the same file can exist in three versions across three folders.
  • Emails — threads with quoted replies and signatures that pollute the actual content.
  • Web pages — with navigation, ads, and boilerplate around the real text.
  • Databases — structured rows that need to be flattened into readable text before they’re useful to an LLM.
  • Tickets and internal knowledge bases (Jira, Confluence, ServiceNow, Zendesk) — semi-structured, frequently updated, and often the highest-value source you have.

Extraction quality caps everything downstream. A layout-aware PDF parser that keeps table structure and reading order will outperform a naive text extractor even if your embeddings and reranker are top-tier. Garbage in, garbage retrieved.

Why Metadata Matters

Metadata is what turns “a pile of text chunks” into “a queryable knowledge base.” At minimum, every chunk should carry:

  • Source — which system and document it came from.
  • Owner — who is responsible for the content, useful for staleness and accountability.
  • Permissions — who is allowed to retrieve it. This is non-negotiable in any multi-tenant or enterprise system; without it, RAG becomes a data leak generator.
  • Page / section — needed for precise citations and for reconstructing surrounding context.
  • Version — so an outdated policy doesn’t get retrieved alongside its replacement.
  • Date — for recency filtering and for answering “as of when” questions.
  • Document type — a runbook, a contract, and a Slack thread should probably be treated differently at retrieval and prompting time.

Metadata is also what makes filtering possible before you ever touch a vector index: “search only in HR documents this user can see, updated in the last 12 months.” That’s a massive relevance and security win that has nothing to do with embeddings.

Chunking Strategy

Chunking is where a lot of RAG quality quietly gets lost. Options, roughly from simplest to most effective:

  • Naive fixed-size chunking — split every N tokens with some overlap. Fast to implement, but it happily cuts sentences, tables, and arguments in half.
  • Section-aware chunking — split along document structure (headings, paragraphs, list boundaries) so each chunk is a coherent unit of meaning.
  • Semantic chunking — group sentences based on embedding similarity so a chunk boundary falls where the topic actually shifts.
  • Parent-child chunking — index small, precise chunks for retrieval, but keep a reference to a larger parent chunk (or the full section) for context assembly. You get precision at search time and enough surrounding context at generation time.
  • Small-to-big retrieval — a related pattern: retrieve small chunks first for precision, then expand to the surrounding parent block before sending it to the LLM.

There’s no universal answer here — it depends on document type. Structured technical docs benefit from section-aware chunking; long narrative content often benefits from semantic chunking; FAQ-style content is usually fine with something closer to naive chunking.

Vector search is excellent at “this means the same thing,” and weak at exact matches — product codes, error codes, acronyms, ticket IDs, config keys. Keyword/sparse search (BM25 and similar) is the opposite: excellent at exact terms, weak at paraphrase and intent.

Production systems run both and combine results — this is hybrid search. It’s not an optimization you add later; it’s close to a requirement once your documents contain identifiers, version numbers, or domain-specific jargon that embeddings don’t represent well.

Retrieval and Reranking

Hybrid search typically returns a wider candidate set — 20 to 100 chunks — because recall matters more than precision at this stage. A reranker (usually a cross-encoder model) then takes the user’s query plus each candidate and scores them for actual relevance, much more accurately than the initial vector/keyword scores. The top 3–8 reranked chunks are what actually go to the LLM.

This two-stage design (cheap broad retrieval, then expensive precise reranking) is the same pattern used in traditional search engines, and it consistently produces better results than trying to get the first-pass retrieval to be perfect on its own.

Context Assembly and Citations

Once you have your top chunks, how you assemble them into the prompt matters:

  • Preserve order and grouping by source document, don’t just concatenate blindly.
  • Include enough metadata inline (source, section, date) so the model can cite it.
  • Deduplicate near-identical chunks from different documents.
  • Explicitly instruct the model to answer only from provided context and to cite the source per claim.

Citations aren’t a UI nicety — they’re your primary tool for catching hallucination and for letting users verify an answer without re-reading the entire source document.

Observability and Evaluation

If you can’t measure retrieval quality, you can’t improve it, and you definitely can’t trust it in production. At minimum, track:

  • Retrieval metrics — recall@k, precision@k, mean reciprocal rank against a labeled evaluation set.
  • Groundedness / faithfulness — does the generated answer actually match the retrieved context, or did the model wander off?
  • Citation accuracy — do the citations actually support the claims attached to them?
  • Latency per stage — extraction, embedding, retrieval, rerank, generation, so you know where time is actually going.
  • Freshness lag — how long between a source document changing and the index reflecting it.

Without this, “the chatbot feels worse this week” is a vibe, not a debuggable signal.

A production-grade setup usually splits into four independently scalable services:

  1. Ingestion service — connects to sources, extracts, cleans, chunks, and enriches metadata. Runs on a schedule or via webhooks/events from source systems.
  2. Indexing service — writes to the vector store and the keyword index, handles re-indexing on document updates and version changes.
  3. Retrieval API — owns hybrid search, filtering by permissions/metadata, and reranking. This is the piece you’ll tune the most over time.
  4. AI answer service — assembles context, calls the LLM, enforces citation formatting, and logs everything for evaluation.

Splitting these apart means you can improve chunking or swap a reranker without touching the generation layer, and you can scale ingestion independently from query traffic.

Common Mistakes

  • Treating chunking as a solved, one-size-fits-all step.
  • Skipping metadata and permissions until “later” — by then it’s a migration, not a feature.
  • Relying on vector search alone and wondering why exact-match queries fail.
  • Sending 20 chunks to the LLM instead of reranking down to the few that matter.
  • Not tracking any retrieval quality metrics until users start complaining.
  • Re-indexing the entire corpus on every small update instead of incremental indexing.

Final Perspective

Production RAG is a data pipeline plus a retrieval system, not just a chatbot feature. The LLM at the end is often the least risky part of the system — it’s reliably good at synthesizing an answer from good context. The hard, valuable engineering work is everything upstream: getting clean text out of messy sources, attaching metadata that enables filtering and trust, chunking in a way that preserves meaning, and retrieving with both precision and recall.

If your RAG system is underperforming, the fix is rarely “try a bigger model.” It’s almost always somewhere in extraction, chunking, metadata, or retrieval. Start there.