RAG in Production: When It's Worth It and When It's Overengineering

In our RAG guide we explained how to make an LLM answer with your company's data: chunk documents, generate embeddings, search by similarity and feed the model only what's relevant. The tutorial works, the demo impresses, and then comes the question that actually matters: shall we take it to production?

And here's the uncomfortable part: in a significant share of the projects we review, RAG wasn't the right answer. Not because the technique fails — it works very well when it fits — but because the problem was solved by something far simpler. This guide is the honest conversation we have with every client before switching on the first vector database.

The Problem: The Demo Lies

A RAG demo is deceptively easy: fifty PDFs, a vector index, a well-chosen question and an answer that looks like magic. Production is a different film:

• The documents aren't fifty clean PDFs: they're ten thousand files with contradictory versions, broken tables and per-department permissions.
• The questions aren't the demo's: they're ambiguous, full of internal jargon, and sometimes have no answer in the documentation — and the model will invent one unless you stop it.
• The answer is no longer there to impress management: an employee uses it to decide, or worse, a customer does.

The distance between "works on my laptop" and "answers in production every day" is where 90% of the work lives — and where you must decide whether the path goes through RAG or somewhere else.

First Things First: Do You Need RAG at All?

Before architecture, the eliminatory question. There are four alternatives that solve problems that look like RAG problems but aren't:

1. Putting the context directly in the prompt. If your relevant documentation fits in the model's context window (and today's windows are huge), you don't need retrieval: paste it whole or by sections. Engineering cost: almost zero.
2. Classic search (full-text, filters, SQL). If the user looks for concrete things — an invoice, an order, an article by its code — a well-written query always beats an embedding. LLMs don't replace the indexes you already have.
3. Fine-tuning. If the problem is one of behaviour (tone, format, response style) rather than knowledge, fine-tuning is the tool. RAG teaches data; fine-tuning teaches manners. Confusing them is the most expensive mistake on this list.
4. An agent with tools. If the answer requires querying live systems (the ERP, the stock API, the CRM), you don't want a static document index: you want the model calling your APIs. That's a different pattern.

RAG comes in when none of the four fits: the knowledge is large, changes frequently, and doesn't fit in the prompt. Extensive technical documentation, support knowledge bases, regulations that get updated, contract repositories. There, RAG isn't one option among many: it's the option.

Signals RAG IS Worth It

1. Large, changing corpus. Thousands of documents updated weekly; reindexing is feasible, retraining is not.
2. Traceability is a requirement. You need to cite the source of every answer ("according to contract X, clause 4.2"). RAG gives you that naturally; fine-tuning doesn't.
3. Per-user permissions. Each user may only receive answers grounded in documents they can access. Filtering at retrieval is feasible; "unlearning" from a fine-tuned model is not.
4. Hallucinations have a real cost. Customer support, legal, healthcare, finance: grounding answers in retrieved documents is the most effective mitigation that exists.
5. The knowledge isn't public. It lives in your wiki, your PDFs, your systems — the model never saw it in training and no amount of prompt engineering will invent it correctly.

Signals It's Overengineering

• Your corpus fits in the prompt. If it's twenty documents that rarely change, a good system prompt with the content gives you the same result for 5% of the cost.
• The problem is format, not knowledge. "I want it to answer in our brand voice" isn't fixed with vectors.
• The data already has structure. If the answer lives in a relational database, write the query (or let the model write it): don't turn perfect rows into fuzzy text chunks.
• You have nobody to maintain it. A RAG in production is a living system: ingestion pipelines, reindexing, quality evaluation. Without an owner, it degrades in silence until someone notices it answers with the 2024 version of the manual.

What Changes in Production (and Tutorials Never Mention)

If you've decided RAG is your pattern, this is what separates the tutorial from the real system:

1. Ingestion with versioning and deletion. Documents get updated and sometimes must disappear (GDPR, terminated contracts). You need to know which chunks come from which document and be able to replace or delete them — "reindex everything from scratch every night" stops being funny at ten thousand documents.
2. Chunking is a product decision. Splitting by fixed size breaks tables and cuts definitions in half. Splitting while respecting structure (headings, sections, whole tables) is the difference between mediocre and good answers, and it demands a parser per document type.
3. Permissions at retrieval, not at response. Access filtering is applied when searching (each user retrieves only what's theirs), never as an instruction to the model to "not mention this". The latter isn't security: it's a suggestion.
4. Hybrid retrieval and re-ranking. Vector search alone fails with codes, acronyms and part numbers. Combining it with keyword search (BM25) and re-ordering results before feeding them to the model improves accuracy more than any prompt change.
5. Continuous evaluation. A set of questions with known answers running in CI and on a schedule: if retrieval precision drops below the threshold, you find out before your users do. Without this, the system degrades in silence.
6. Latency and cost per answer. Every answer pays: search + re-ranking + generation with long context. Measuring it per query — and caching frequent questions — is what decides whether the project is sustainable or an expensive demo.

Common Mistakes

1. Starting with the vector database. Choosing between pgvector, Qdrant or Pinecone is the least important decision of the project; ingestion and chunking quality is the most important one.
2. Not measuring retrieval separately. If an answer is bad, did the search fail or the generation? Without separate metrics, every fix is a lottery.
3. Trusting security to the prompt. "Do not reveal confidential information" written in the system prompt is not an access control.
4. Indexing garbage. If 40% of the corpus is outdated or duplicated, 40% of the answers will lean on garbage. Corpus curation is business work, not engineering.
5. Promising accuracy without evaluation. "It works pretty well" is not a metric. An evaluation set of 50 real questions, built before launch, is worth more than any third-party benchmark.

In Summary

RAG is the right pattern when knowledge is large, changes frequently, requires citations and permissions, and doesn't fit in the prompt. It's overengineering when a query, a good prompt or an agent with tools solves the same problem. And when it fits, production demands what tutorials leave out: versioned ingestion, structure-aware chunking, permissions at retrieval, hybrid search and continuous evaluation. The demo is 10%; the rest is serious engineering.

If you're considering taking AI with your own data to production — or already did and the answers aren't up to standard — it's exactly the kind of system we design and audit. Tell us about your case →

Keep reading: RAG: How to make an LLM answer with your company's data.