Ask most developers what RAG (Retrieval-Augmented Generation) means, and you’ll get some version of the same answer: put your documents in a vector database, embed the query, pull back the top-k matches, and hand them to an LLM. Done.
That description isn’t wrong. It’s just incomplete enough to be misleading.
Say you’re building an internal support chatbot, and someone asks it: “Can I get a refund for a digital product purchased more than 30 days ago?”
For the bot to answer this correctly, it doesn’t just need “a relevant document.” It needs the current refund policy, not last year’s PDF still sitting in the index. It needs the clause about digital goods specifically, which might live in a separate metadata field entirely. And if the wrong chunk gets retrieved, the LLM will still generate a fluent, confident, completely wrong answer.
That’s the real engineering question behind RAG: how do you make sure the model receives the right information before it starts generating anything at all? The vector database is one component in that pipeline, rarely the reason a RAG system fails, and rarely the reason one succeeds either.
The Pipeline, Not the Database
A production RAG system is really a sequence of engineering decisions, each with its own failure modes: ingestion, chunking, embedding, indexing and metadata, retrieval, reranking, and generation.
Most public tutorials spend 90% of their time on embedding and retrieval, treating everything else as boilerplate. In practice, chunking, metadata, and reranking are where most reliability problems actually originate.
Chunking: The Decision Everyone Underestimates
How you split a document determines what’s even possible to retrieve later. Chunk too large, and relevant text gets diluted inside pages of unrelated content. Chunk too small, and you strip away the context that makes a passage interpretable.
Fixed-size chunking is simple and often good enough for homogeneous content. Structure-aware chunking, splitting along headings or list items, tends to perform better on real documentation. Semantic chunking, where boundaries follow shifts in meaning between sentences, can outperform both on unstructured content, at the cost of extra preprocessing. There’s no universally correct strategy; the right choice depends on document structure and query patterns.
Fixed-size, structure-aware, and semantic chunking produce very different retrieval units from the same document.
Embeddings: Similarity Isn’t the Same as Relevance
Embedding models place semantically similar text close together in vector space, but that’s not the same as relevance to a specific query. Two passages can be topically similar while only one actually answers the question.
Embedding models also tend to struggle with negation and numeric thresholds. “Refunds available within 30 days” and “refunds not available after 30 days” can end up with nearly identical embeddings, because the surrounding vocabulary is almost the same. If your chatbot depends on getting that distinction right, similarity alone can’t be trusted to carry it.
Retrieval and Reranking: Precision Over Recall
A vector search returning the top 20 candidates is optimizing for recall, making sure the right answer is somewhere in the set. But an LLM’s context window is a poor place to make the model do the sorting itself; irrelevant chunks add noise, cost, and sometimes measurably worse answers.
A cross-encoder reranker looks at the query and each candidate together, rather than independently, producing a far more precise relevance score. It’s more expensive than vector search, which is why it’s applied to a small candidate set rather than the whole index, cheap, high-recall retrieval followed by expensive, high-precision reranking.
High-recall retrieval followed by high-precision reranking narrows candidates before they reach the model.
Metadata Filtering and Where RAG Actually Fails
Semantic search alone can’t distinguish “the 2024 policy” from “the 2026 policy” if both use nearly identical language. Structured metadata, effective date, version, product category, access level, does work embeddings can’t. Filtering before or during vector search narrows the candidate pool to documents that are actually eligible to answer the query.
In practice, failures cluster around a handful of patterns: stale documents never removed from the index, chunks that lose critical context once separated from their source, queries phrased in ways that don’t embed close to the answer, and generation-time hallucination where the LLM ignores the retrieved context entirely. Most of these failures happen upstream of generation, when a RAG system gives a wrong answer, the model was often never given the right material to work with.
Designing for Production, Not for Demos
A demo RAG pipeline can get away with a single vector index and a static document set. A production system usually needs incremental re-indexing as documents change, metadata schemas that support filtering, a reranking stage, and monitoring for retrieval quality, not just generation quality.
A production RAG system includes incremental re-indexing, metadata support, reranking, and monitoring, not just a vector store and an LLM.
None of this is exotic engineering. It’s closer to building a good search system than to prompt engineering, which is precisely the point. Treating RAG as “vector database plus LLM” undersells the retrieval engineering that determines whether the system is trustworthy. Get the pipeline right, and the LLM’s job becomes much simpler: synthesize an answer from context that’s actually correct.
You may also like : Exactly-Once Semantics in Apache Kafka: A Guide
