Ways of implementing RAG

Why you should care, even if you never write this code

Somebody asks me every few weeks: “LangChain or LlamaIndex?” — as if one of them were the right answer. Neither is. What matters is knowing what each option decides for you, because every decision a library makes on your behalf is a decision you will later have to live with or un-make.

Two minutes from now you will be able to read a complete RAG pipeline from top to bottom — even if you barely code — and you will know which of the five ways to build it fits your situation. That is worth more than any framework name.

A few words I will use without apology: a vector DB is a database that stores embeddings — lists of numbers that capture meaning — and finds the nearest ones to your question (how that works). A framework is a box of ready-made building blocks: loaders, splitters, retrievers. Convenient, and opinionated.

Five ways to build the same pipeline

The RAG pipeline from the last article — chunk, embed, index, retrieve, assemble, generate — can be built at several altitudes. From lowest to highest:

• Fully manual — embeddings API or a local model, numpy, your own prompt strings. Total control, total responsibility.
• Raw SDKs — the model provider’s SDK plus a vector DB client. The standard production choice: minimal magic, easy to reason about.
• LangChain — the general-purpose LLM framework. Huge ecosystem, many integrations, abstraction layers several levels deep.
• LlamaIndex — a framework focused specifically on data ingestion and retrieval. Often the fastest path to a good RAG prototype.
• Haystack — pipeline-oriented framework from deepset, popular for production search and QA systems with strong typing.

Every option implements the same six steps. The differences are how much they hide, how much they decide for you, and how hard it is to leave:

None of them is “more advanced” than the others. They are different amounts of decisions someone else already made for you. The honest question is never “which is best” but “which of these pre-made decisions do I want to inherit?”

The manual version, end to end

Here is the whole idea compressed into two small files: a mock embedding model and a mock LLM (no API keys needed — the mechanics are what matter), then the RAG loop itself:

1# Tiny stand-ins for a real embedding model and LLM.
2# Swap these two functions for real API calls and nothing else changes.
3import math
4 
5def embed(text):
6  # A real embedding model turns text into a long list of numbers
7  # (hundreds or thousands of them) that captures its meaning.
8  # We fake it with 4 numbers: how often the words "refund",
9  # "password" and "plan" show up, plus the length of the text.
10  # Enough to see the mechanics; useless for real search.
11  words = text.lower().split()
12  return [
13      words.count(#98c379">"refund") / (len(words) + 1),
14      words.count(#98c379">"password") / (len(words) + 1),
15      words.count(#98c379">"plan") / (len(words) + 1),
16      len(words) / 100,
17  ]
18 
19def llm(prompt):
20  # A real LLM reads the prompt and writes an answer.
21  # Ours only prints what it was handed - so you can see exactly
22  # which chunks reached the model. That is the whole game of RAG:
23  # control what lands here and you control the answer.
24  for line in prompt.splitlines():
25      if line.startswith(#98c379">"[") or line.startswith("Q:"):
26          print(line)
27  return #98c379">"Answer grounded in the chunks above."
28 
29def cosine(a, b):
30  # "Cosine similarity": how close are two lists of numbers?
31  # 1.0 means near-identical, 0.0 means unrelated.
32  # This is how retrieval decides which chunks survive.
33  dot = sum(x * y for x, y in zip(a, b))
34  na = math.sqrt(sum(x * x for x in a))
35  nb = math.sqrt(sum(x * x for x in b))
36  return dot / (na * nb) if na and nb else 0.0

Read the two tabs in order: mocks.py is the cast (a fake embedding model and a fake LLM), rag.py is the play.

If code is not your language, here is the same story in plain words:

1. Three sentences stand in for your company help center.
2. Indexing (once, offline): each sentence is converted into a list of numbers — an embedding — that roughly captures its meaning. You keep the sentence and its numbers. This is your searchable index.
3. The question goes through the exact same conversion. Same ruler on both sides — that is why it works.
4. Retrieval: compare the question’s numbers with every chunk’s numbers, keep the three closest. Nothing is “understood” here; it is arithmetic on lists of numbers.
5. Assembly: paste those chunks above the question — like handing someone a page of notes and saying “answer using this, nothing else.”
6. Generation: the model reads the notes and writes the answer.

That is RAG. Six moves, none of them clever on its own.

In production, embed and llm become real API calls and the sorted list becomes a vector database query — but the shape of the code does not change. That is the point: RAG is not hard, it is just a pipeline. And the pipeline is only as good as step 4 — if retrieval hands the model the wrong three chunks, no framework and no model will save the answer.

The frameworks, honestly

LangChain gets you running in minutes: document loaders, splitters, vector store wrappers, chains. The trade-off is depth of abstraction — when retrieval quality is off, you may find yourself debugging a framework instead of your pipeline. Its strength is breadth: one API over dozens of providers.

LlamaIndex starts from the data side. Indexes, query engines, and sensible RAG defaults are the core product, not an afterthought. For “chat with these 500 PDFs” it is often the fastest route. The cost is the same class as LangChain: you adopt its abstractions.

Haystack leans toward production search systems: explicit pipelines, typed components, cleaner to test. Smaller community than LangChain, but a more predictable surface.

One honest note on all three: they move fast and they break things. Pin your version, read the changelog before upgrading, and keep your own thin wrapper around the few calls you make — so a breaking change is one file, not thirty.

How to choose

A decision rule that has aged well:

1. Learning the concept? Build it manually once, like the example above. You will never again wonder what a retriever abstraction is hiding.
2. Shipping a focused product? Raw SDKs plus a vector DB client. Most production RAG systems are exactly this — a few hundred lines you fully understand.
3. Prototyping over messy documents at speed? LlamaIndex or LangChain. Accept the abstractions to buy days of time.
4. Large search/QA platform with a team? Haystack’s structure earns its keep.

Where this leaves you

You can now build RAG five different ways and know the trade-off of each: control versus convenience versus concepts. One thread ties all of it together: everything the model sees — system prompt, retrieved chunks, the question — is context you engineered. Managing that deliberately is its own skill.

If any of the six steps still feels foggy, walk the RAG pipeline article again — it is the same six steps, one at a time.