What is RAG

Two kinds of exam

Picture two exams.

In a closed-book exam you walk in with only what you memorized. Ask about something you never studied and you have two options: leave it blank, or write something plausible and hope. Language models are built to never leave a blank — so they write something plausible. That is a hallucination: a confident answer that sounds perfect and is simply made up.

In an open-book exam you walk in with the textbook. You do not need to have memorized the refund policy; you need to be able to find the right page and read it. The exam did not get easier because you got smarter. It got easier because the information was in the room.

RAG — Retrieval-Augmented Generation — is the open-book exam. Before the model answers, your code fetches the pages that matter and slides them onto the desk. The model then answers by reading, not by recalling.

That is the whole idea. The rest of this article makes each half precise: what retrieval actually is, and why the model cannot simply know your data.

What “retrieval” means

Retrieval is the ordinary thing software has always done: given a question, find the passages that probably contain the answer. It is search. Not thinking, not understanding — search across your own documents, returning text.

It takes your question and hands back a handful of passages:

Question: “What is our refund window for annual plans?”

Retrieved passage: “Annual plans can be refunded in full within 14 days of purchase. After 14 days, we issue a pro-rata credit.”

That passage lives in a file called refunds.md in your company wiki. The model has never seen it. Retrieval found it, and we paste it into the prompt so the model can read it before answering.

Three words of vocabulary, because everything else builds on them:

• The context window — the model’s working memory for one conversation. Everything you paste in and everything it writes back has to fit inside it.
• Augmented — the prompt was topped up with those retrieved passages.
• Generation — the model writing the answer.

So retrieval-augmented generation is just: find, paste, then write.

Why the model can’t just know your data

This is the part people resist, so here it is plainly: a model cannot look anything up. It has no database to query, no internet connection during the call, and no private storage where your files live. It receives one block of text and predicts what comes next. That is the entire mechanism.

So everything it can possibly “know” at the moment it answers comes from exactly two places:

1. Its weights — the patterns compressed into it during training. Frozen the day training ended. You cannot edit one fact in there; you would have to retrain.
2. The context window — the text you sent along with this request. This is the only half you control.

Your data sits in neither place by default, for two separate reasons:

• The cutoff date. Training ended on some date. The model has never heard of last week’s pricing change, today’s outage, or the refund policy you rewrote yesterday.
• It was never in the training data at all. Your docs, tickets, contracts and internal wiki are private. Nobody at OpenAI read your company wiki, so there is nothing in the weights to recall — no matter how well you phrase the question.

And here is why that produces confident nonsense instead of silence. The model is not checking a fact store and coming up empty. It is always predicting the next word. Given “Our refund window for annual plans is ___”, the most likely continuation is a number and a plausible sentence, because that is what such sentences look like on the internet. It fills the blank because filling blanks is the only thing it does.

The old hope was “maybe it memorized enough”. RAG replaces hope with a mechanism: put the fact in the prompt, and there is nothing left to guess.

Seeing the difference

Here is the same question in a closed-book exam — a model with no access to your data — and then with RAG. Watch what happens when we give it something to read:

The first answer sounds confident and is wrong — that combination is called a hallucination. The model isn’t lying; it’s pattern-completing from generic internet text, because that’s all it has. The second answer cites your actual policy, because it was sitting right there in the prompt.

Why not just paste everything into the prompt?

Every beginner asks this, and it’s a fair question: if the model is smart, why not hand it the whole wiki at once and be done?

Three reasons, in order of how much they’ll hurt you:

1. It doesn’t fit. The context window is a hard budget. 200k tokens sounds enormous until you count: a decent-sized company wiki is millions of tokens. You physically cannot paste it all.
2. It’s slow and expensive. Every token you send costs money and adds latency — on every single question. Paying to re-read 500 pages to answer “what’s the refund window?” is a bad deal.
3. It makes answers worse. This is the surprising one. Bury the one relevant paragraph inside 500 pages and the model gets distracted: it hedges, mixes up two policies, or answers confidently from the wrong section. More context is not more accuracy — the right context is.

RAG is that filter. A search step finds the handful of paragraphs that actually matter, and the prompt carries only those. You get the knowledge of a million documents for the price of a few thousand tokens.

Why hallucinations happen without it

Three forces push a model toward making things up:

• No access. The fact simply isn’t in the training data or the prompt. The model’s only move is to produce plausible-sounding text anyway.
• Instruction pressure. “Answer the question” is a strong instruction. A model that always replies with “I don’t know” is a bad assistant, so it errs on the side of answering.
• Fluency. LLMs are optimized to produce fluent text, and fluent text about a refund policy looks the same whether the facts are real or invented.

RAG doesn’t fix the model. It changes the situation: instead of recalling, the model quotes. Reading is something LLMs are genuinely good at.

Try RAG by hand, right now

You don’t need a database to feel how different this is. Copy the prompt below into ChatGPT, paste any document you have — a contract, a manual, a long email — where it says DOCUMENT, and ask it a question you already know the answer to. Then ask the same question in a fresh chat without the document and compare:

Here is a document:
---
DOCUMENT
---

Answer my question using ONLY the document above.
Rules:
- If the answer is not in the document, say "I don't know" and say what
is missing.
- Quote the exact sentence you based the answer on.
- Do not use anything you know outside the document.

Question: ...

That’s RAG in its purest form — you did the retrieval by hand. Everything in the rest of this track is about automating the part you just did with copy-paste.

Where RAG is used

Once you see the pattern — retrieve, then generate — you notice it everywhere:

• Support bots that answer from your help center instead of inventing policies.
• Internal knowledge assistants over wikis, tickets, and Slack archives.
• Code assistants grounded in your actual codebase, not generic tutorials.
• Legal and finance tools that must cite exact contract clauses.
• Chat-with-your-docs — the classic demo that made RAG famous.

If a product needs to be right about specific information that changes over time, it’s almost certainly RAG-shaped.

RAG vs fine-tuning

The classic point of confusion. Fine-tuning means continuing the model’s training on your own examples, so the new behavior becomes part of the model’s weights. Both approaches inject knowledge, but they do it at completely different layers:

This is the exam analogy from the top, applied: fine-tuning is months of study before the exam; RAG is being allowed to bring the book into it. Fine-tuning teaches skill, RAG provides materials. If the question is “does the model know the new refund policy”, RAG wins by a mile — the policy changed last week, and retraining over one paragraph would be absurd.

Where fine-tuning genuinely wins, and it’s worth being honest about it:

• Tone, format and habits. Always answering in your exact JSON shape, writing like your support team, refusing in your words. That’s behavior, not facts — RAG can’t teach it.
• Latency and cost at scale. Baked-in behavior means shorter prompts, which means fewer tokens per call. At millions of calls this is real money.
• Small models. A tiny model with the right fine-tuning can beat a big model with a long prompt on a narrow, repetitive task.

And where RAG honestly struggles: it’s only as good as its retrieval. If the search step returns the wrong paragraph, the model will answer fluently from the wrong paragraph — a different flavor of the same problem. RAG reduces hallucinations dramatically; it doesn’t delete them.

They aren’t rivals. Serious products often do both: fine-tune for tone and format, RAG for facts.