The RAG pipeline, step by step

Two phases, one pipeline

RAG has two lives. Indexing happens offline: whenever your documents change, you cut them into chunks, embed them, and store them in a vector database. Querying happens online, per user question: embed the question, retrieve the nearest chunks, stuff them into the prompt, and let the model answer.

Step through the full loop:

Each step hands a concrete artifact to the next: text becomes chunks, chunks become vectors, vectors get stored, a question pulls five chunks back out, and those five chunks plus your rules plus the question become one prompt.

The rest of this article zooms into the two stages engineers get wrong most often: chunking, and assembling that final prompt.

Why not skip all this and paste the whole document?

Because pasting everything fails in three different ways at once:

• It doesn’t fit. A context window is a hard budget. A 50-page PDF is roughly 25–40k tokens; a real knowledge base is millions. There is no window big enough for “everything”.
• It’s expensive on every question. You pay for those tokens on every call, and you wait for them to be read. Answering “what’s the refund window?” by re-reading 500 pages is a terrible deal at scale.
• It makes answers worse. Drown the one relevant paragraph in 500 pages and the model hedges, mixes two policies, or answers confidently from the wrong section. The right few chunks beat the wrong five hundred pages.

That is what the pipeline buys you: a filter that turns a million documents into the handful of paragraphs this question actually needs.

Chunking: the retrieval unit

Why chunk at all? Two reasons. First, precision: a 50-page PDF contains answers to hundreds of different questions; retrieving the whole thing for each one floods the prompt with noise. Second, embeddings blur on long texts — a single vector for an entire manual averages away everything specific about any one section.

Chunk size is a real trade-off:

• Too small (a sentence or two) — chunks lose context. “The refund is 14 days” means nothing without knowing it’s about annual plans.
• Too large (many pages) — each chunk covers many topics, so its vector matches everything and nothing, and you pay for tokens you don’t need.

The practical starting point: 300–800 tokens per chunk, with 10–20% overlap between consecutive chunks so ideas that straddle a boundary don’t get lost. Split on natural structure — headings, paragraphs, list items — not on a raw character count.

Prompt assembly: the token budget

After retrieval you hold three ingredients: instructions, the retrieved chunks, and the user question. Here is what a realistic RAG prompt looks like token-wise:

Notice how small this actually is. A common misconception is that RAG means cramming huge documents into the context window. It’s the opposite: RAG is how you answer questions about a million documents while sending only a few thousand tokens. The vector database is the filter; the prompt only ever sees the shortlist.

Here is the shape of that final prompt as a template you can copy and adapt — this is the part that decides whether your bot says “I don’t know” or invents a policy:

You answer questions using only the numbered sources below.

Rules:
- Use ONLY the sources. If the answer is not there, say
"I don't have that in our docs."
- Cite the source number you used, like this: [1].
- Never mix facts from different sources into one claim.

Sources:
[1] <chunk 1 text>
[2] <chunk 2 text>
[3] <chunk 3 text>

Question: <user question>

Answer in at most 3 sentences.

Two details in that template do most of the work: “if the answer is not there, say so” (it gives the model permission to admit ignorance instead of guessing) and “cite the source number” (it makes every claim checkable, by a human or by your tests).

Quality lives and dies at retrieval

The model can only be as good as the chunks you give it. In practice:

• Bad chunks in, confident nonsense out. If retrieval returns the wrong section, the model won’t object — it will answer from your wrong context fluently.
• Measure retrieval separately. Before blaming the LLM, check whether the right chunk was even in the top 5. This is a search-quality problem with search-quality metrics.
• Upgrade later: hybrid search (vectors plus keyword matching), re-ranking the top 50 candidates with a cross-encoder, metadata filters (“only 2024 documents”).