Information in this post reflects publicly available sources as of October 16, 2026.
You ask the company's new AI assistant a simple question: what is the retry policy for the payments API? It answers instantly. Three retries, exponential backoff starting at one second, idempotency keys on every request. It sounds exactly right.
It is also invented. Your payments API retries twice, with a fixed 500 millisecond delay, and the reasoning lives in a design doc on the internal wiki. The model has never seen that doc. It gave you the most plausible answer it could assemble from everything it learned in training, which is a polite way of saying it guessed.
The fix most teams reach for is retrieval-augmented generation, or RAG. Before the model answers, you find the passages in your own documents that are relevant to the question and put them in the prompt. The model stops guessing and starts reading.
A model only knows what it was trained on
A model's knowledge has two hard edges. It stops at the training cutoff, so anything that changed since then is missing. And it never included your private data in the first place: your wiki, your tickets, your contracts, your codebase. For most business questions, the answer lives on the far side of both edges.
There are three broad ways to close that gap. You can fine-tune the model on your documents, which is expensive, slow to update, and surprisingly bad at teaching specific facts. You can paste every document into the prompt, which works until the corpus outgrows the context window or the bill. Or you can retrieve just the relevant pieces at question time, which is what RAG does.
The idea is simple enough to fit in a sentence: search first, then answer. Almost all the engineering effort goes into making the search part good.
How it works
Two pipelines
A RAG system is really two pipelines that meet at a shared index. The indexing pipeline runs offline whenever documents change. It splits documents into pieces, converts each piece into a searchable form, and stores the results.
The query pipeline runs on every question. It searches the index, picks the best pieces, and hands them to the model along with the question.
Chunking
Documents are too long to retrieve whole, so the indexing pipeline splits them into chunks, typically a few hundred tokens each. Chunk size is a real trade-off. Small chunks match questions precisely but lose surrounding context. Large chunks keep context but dilute the match and eat more of the prompt.
Splitting on structure works better than splitting on character count. Break at headings, paragraphs, and function boundaries rather than mid-sentence, and keep tables intact. A common upgrade is to prepend each chunk with a short header naming the document and section it came from, so a chunk that says "the limit is 500 requests" still makes sense when it arrives alone.
There is no universally right chunk size. A reasonable starting point is a few hundred tokens with a small overlap between neighboring chunks, so a sentence that straddles a boundary appears in both. From there, tune against real questions rather than intuition, because the best size depends heavily on how your documents are written.
Embeddings
Each chunk is then passed through an embedding model, which turns text into a list of numbers called a vector, often with a thousand or more dimensions. The useful property is that texts with similar meaning end up with vectors that sit close together, even when they share no words. "How do I reset my password" lands near "steps to recover account access."
At query time, the question is embedded with the same model, and search becomes geometry: find the stored vectors nearest to the question's vector. Vector stores use approximate nearest neighbor indexes, most commonly HNSW, to do this across millions of chunks in milliseconds.
One operational detail catches teams out. Vectors from different embedding models are not comparable, so switching to a newer embedding model means re-embedding the entire corpus. Store the model name and version alongside every vector, and plan for a full reindex whenever you change it.
The diagram shows the catch: embeddings capture similarity, not relevance. A chunk about the email service's retry policy is about as close to the question as the payments chunks are, because the wording and topic overlap. Embedding search will happily return it, and the model may happily use it. Much of the craft in RAG is about telling those two apart.
Hybrid search and reranking
Pure vector search has a known blind spot: exact terms. Error codes, product SKUs, function names, and acronyms carry meaning that embeddings blur. A question about ERR_PAY_4021 wants the document containing that exact string, not documents about payment errors in general.
So most production systems run hybrid search: a vector search and a classic keyword search (usually BM25, the algorithm behind traditional search engines) side by side, with the two result lists merged. The merge is often done with reciprocal rank fusion, which rewards chunks that rank well in either list.
Then comes a reranker. Embedding search compares precomputed vectors, which is fast but shallow.
A reranker reads the question and each candidate chunk together and scores how well that chunk answers that question. It is too slow to run over the whole corpus, but fast enough to run over fifty candidates. Retrieve broadly, then rerank narrowly.
Putting it in the prompt
The last step is ordinary prompting. The query pipeline, with hybrid search, permission filtering, and reranking, looks roughly like this:
def answer(question, user):
candidates = index.search(
vector=embed(question),
keywords=question, # hybrid: vector + BM25
filter={"allowed_groups": user.groups}, # enforce access at query time
top_k=50,
)
top = reranker.rank(question, candidates)[:5]
sources = "\n\n".join(
f"[{i + 1}] {c.title} ({c.url})\n{c.text}" for i, c in enumerate(top)
)
prompt = f"""Answer the question using only the sources below.
Cite sources by number. If the sources do not contain the answer, say so.
{sources}
Question: {question}"""
return model.generate(prompt)The index, reranker, and model objects stand in for whichever products you use, but every production pipeline has these steps. Note the filter: permissions are enforced inside the search, so a user can never retrieve a chunk they could not open themselves. Note the instructions too. Asking for citations makes answers checkable, and explicitly allowing "the sources do not contain the answer" is what stops the model from falling back to guessing.
Agentic RAG
The pipeline above retrieves once, then answers. That fails on questions that need several lookups, like "which services changed their retry policy after the March incident?" The model needs to find the incident report first, then search for each service it mentions.
The answer is to give retrieval to an agent as a tool. The model decides what to search for, reads the results, and searches again until it has enough.
This is agentic RAG, and it is simply the agent loop from an earlier post in this series with a search tool attached. Many teams now expose their search index as an MCP server so any agent can use it. The trade-off is the usual one for agents: better answers on hard questions, at the cost of more calls, more latency, and less predictability.
When to use RAG, and when not to
RAG is one of several ways to get the right information in front of a model. The right choice depends on the shape of the data and the question.
| Approach | Good for | Weak at |
|---|---|---|
| RAG | Large, changing, or private corpora; answers that need citations; per-user access control | Questions that need the whole corpus at once; only as good as its retrieval |
| Long context | Small, stable corpora; reasoning across several whole documents | Cost and latency on every query; per-user data that defeats caching |
| Fine-tuning | Tone, format, and domain-specific behavior | Adding facts reliably; keeping them current |
| Tools and SQL | Counts, totals, filters, and live data | Unstructured text |
Long context deserves a closer look, because it changed the default. Several frontier models now accept a million tokens or more, and prompt caching makes repeated prefixes cheap. If your entire knowledge base is a 100-page handbook that rarely changes, put the whole thing in the prompt, cache it, and skip RAG entirely. Retrieval earns its complexity when the corpus is too big, changes too often, or differs per user.
Watch for questions that look like search but are really queries. "How many P1 incidents did we have last quarter?" cannot be answered by retrieving five similar chunks. It needs a count over structured data, which means a SQL query or an API call, not RAG.
What goes wrong
Retrieval misses, and the model answers anyway. When a RAG system gives a wrong answer, the cause is usually that the right chunk never reached the prompt. Teams often tune the prompt or swap the model first, which fixes nothing. Debug retrieval before generation: log what was retrieved for every bad answer, and check whether the right passage was there.
Chunks lose their context. A chunk that reads "this limit was raised to 1,000 in v2" is useless if you cannot tell which limit or which product. Tables split across chunks lose their headers. Structure-aware splitting and contextual headers on each chunk fix most of this.
Stale and conflicting documents. The index contains last year's policy and this year's, and the retriever has no idea which is current. Store dates and status as metadata, filter out superseded documents, and reindex when sources change rather than on a quarterly whim.
Permission leaks. If the index ignores access control, the assistant becomes a way to read documents you are not allowed to open. Enforcing permissions in the search filter, as in the code above, is the only reliable fix. Filtering after the model has seen the text is too late.
Retrieved text can carry instructions. A wiki page or support ticket can contain text written to steer the model, and RAG delivers it straight into the prompt. Treat retrieved content as untrusted data, especially in agentic RAG where the model can also take actions.
The common thread is measurement. Build a small golden set of real questions with known correct sources, and track two numbers separately: did retrieval find the right chunk (often reported as recall@k), and did the model answer correctly given what it found. Without that split, you cannot tell which half of the system to fix.
Cost, latency, and complexity
The economics of RAG are front-loaded. Embedding is cheap: at an illustrative $0.10 per million tokens, embedding a 50-million-token document corpus costs about $5. The real indexing cost is engineering time spent on parsing, chunking, and keeping the index in sync with its sources.
At query time, RAG keeps prompts small. Five chunks plus instructions come to around 4,000 input tokens. At an illustrative $3 per million input tokens, that is about $0.012 per question.
Sending a 200,000-token knowledge base instead costs about $0.60 per question before caching, fifty times more. Caching narrows the gap sharply for stable shared content, which is exactly why long context wins on small, stable corpora.
Latency follows the funnel. Embedding the question and searching the index usually take tens of milliseconds, and a reranker adds roughly a hundred more. Generation still dominates, and smaller prompts generate faster. The complexity cost is the one teams underestimate: a RAG system is a search engine plus a data pipeline plus a model, and each part needs monitoring.
The current landscape
For storage, the first question is whether you need a dedicated vector database at all. The pgvector extension adds vector search to Postgres, and with the 0.8 release line it handles filtered queries far better than earlier versions. For many teams with a few million chunks, that is enough, and keeping vectors next to relational data makes permission filtering straightforward.
Dedicated options cover larger scale and more advanced retrieval. Pinecone is fully managed, while Qdrant and Weaviate can be self-hosted or run in the cloud, and both support hybrid search natively. turbopuffer stores vectors on object storage, which suits large multi-tenant workloads. Elasticsearch, OpenSearch, and MongoDB Atlas have also added vector search alongside their existing keyword search.
Embedding models come from OpenAI, Google, Cohere, Voyage AI, and a range of strong open-weight families. Cohere and Voyage AI also offer hosted rerankers, and open-source cross-encoders work well when you want to self-host. Frameworks such as LlamaIndex and LangChain package the pipeline, and the major clouds offer managed options, including OpenAI's file search, Google's Vertex AI Search, and Amazon Bedrock Knowledge Bases, which trade control for speed of setup.
Summary
RAG looks like an AI technique, but it is mostly a search problem with a model attached at the end. The model's job is the easy part: read a few relevant passages and write an answer. Whether those passages are the right ones is decided by chunking, embeddings, keyword matching, reranking, and permissions, all of which are classic information retrieval.
That reframing changes where you spend effort. When a RAG system gives a bad answer, look at what it retrieved before you touch the prompt or the model. Measure retrieval on its own. And before building any of it, check whether your corpus simply fits in the context window, because the best retrieval pipeline is sometimes the one you do not need.
Part of the Applied AI series: AI from the builder's side.