All lessons · 44
How LLMs Actually Work
What a Large Language Model really is: next-token prediction, tokens, training vs inference, the context window — and why models sometimes confidently make things up.
A very good text prediction machine
Why should you care? Because once you see the trick, the magic becomes predictable — and you stop being surprised by the model’s failures. Every weird thing an LLM does, from inventing a citation to “forgetting” your name, comes out of the one mechanism on this page.
Strip away the hype, and a Large Language Model does exactly one thing: given some text, it predicts what comes next. That’s it. That’s the whole trick.
Ask it “The capital of France is” and it computes probabilities for every possible next word: “Paris” — very likely, “the” — possible, “Banana” — extremely unlikely. It picks one, appends it to the text, and repeats. Answer by answer, paragraph by paragraph, it’s still just doing the same thing over and over: predict the next piece, add it, predict again.
What makes it feel magical is scale. During training the model was shown an enormous amount of text, and it turns out that predicting the next piece well requires something that looks a lot like understanding — grammar, facts, code, even step-by-step reasoning.
So let’s be precise about what is not in that loop:
- It does not look anything up. There is no database it queries while answering. Facts live in its parameters — the numbers it learned during training — and come back out as predictions.
- It does not think the way a person does. “Reasoning” in an answer is also just predicted text. It often helps, and it is not a proof of anything.
- It does not check its own answer. Nothing in the mechanism compares the result against reality. That part is your job.
Inference: predicting the next token
Running a trained model is called inference. Each time you send a message to an LLM, it predicts your answer one piece at a time.
Here’s the fun part: the model doesn’t just pick the single most likely word. It samples — rolls weighted dice — from a probability distribution. A single setting called temperature controls how adventurous those dice are. Think of it as a dial: at low temperature the model almost always takes the safest word; turn it up and less likely, weirder words start winning. Try it:
low temperature → the sharp token wins almost always (safe, repetitive). high temperature → the distribution flattens, unusual tokens get a real chance (creative, risky).
Click “sample” a few times at low temperature: you’ll get “Paris” every time. Drag the slider up and suddenly “Lyon” or even stranger choices appear. This is why the same prompt can give you different answers — and why a low temperature is what you want for factual tasks, while a higher one can help with brainstorming.
Why this matters to you
Almost every “weird model behavior” you’ll debug as an AI engineer traces back to this mechanism. The model is not looking up answers — it is rolling weighted dice, one token at a time.
Tokens: models don’t see words
One surprise for most people: an LLM doesn’t work with words. It works with tokens — pieces of text that can be a whole word, a part of a word, or even just punctuation.
Roughly: common English words are usually one token, rarer words get split (“astounding” might become “ast” + “ounding”), and numbers often get chopped into digits. The word “unbelievable” might be three tokens; the space before a word counts too.
This explains a lot of practical quirks:
- Models can struggle to count letters in a word — they never see letters, only tokens
- Non-English text often costs more tokens (and more money) per sentence
- API pricing and context limits are always measured in tokens, not words
A loose rule of thumb: one token is about three-quarters of an English word. Talking to a model: tokens, context and sampling digs much deeper into all of this.
Training vs inference
Two phases, completely different in cost and in what they change:
- Training — the expensive, one-time (per model) phase. The model is shown enormous amounts of text and gradually adjusts its internal parameters — billions of numbers — to get better and better at predicting the next token. This takes months on thousands of GPUs (the expensive chips these models run on) and is done by model labs, not by you.
- Inference — what happens every time you use the model. Parameters are frozen; nothing is learned from your chat. The model just runs forward, token after token.
This has a consequence people find counterintuitive: the model doesn’t remember you. Close the chat, start a new one, and it knows nothing about your previous conversation. Everything it “knows” about you has to fit into what you send it — which brings us to the context window.
The context window
The context window is the model’s working memory: all the text it can consider at once. Think of it as the model’s desk — everything it is allowed to look at right now has to be lying on that desk. On it goes the system prompt — the standing instructions the app’s developer wrote for it, which you never see — plus your messages, files you attach, and its own replies. Everything must fit in this window, and when it fills up, older content has to be dropped or summarized.
Watch a realistic session fill the window — a coding assistant answering a question about a file:
A few things to notice in that simulation:
- The system prompt is always there, even though the user never sees it
- The attached file costs more tokens than everything else combined
- Every new turn re-sends the whole history — that’s why long conversations get slower and more expensive as they go
Managing what goes into this window — and what stays out — is such an important skill that it has its own name and its own article: context engineering. There’s a lot of “later articles” in this track, and that’s on purpose: each idea here gets its own deep dive.
Seeing it turn by turn
One more way to feel how a conversation works. Each message you send makes the whole conversation longer — the model re-reads everything every time:
The model “remembers” your name only because your first message is still in the context window and gets re-sent with every turn. In a fresh chat, it genuinely has no idea who Dana is. Memory across sessions is a feature you build yourself — with databases, not magic.
Everything is this one call
Here is the insight that makes the whole roadmap click: chat, agents, tools, RAG — all of it is this same one call, repeated with different text.
The chat window in your favorite app is just a loop: your message plus all previous messages go in, the continuation comes out, it gets appended to the conversation, and the whole thing is sent again on your next turn. That is the entire chat UI — nothing more.
Everything else you will meet in this track is the same single call with different text packed into the context:
- Tools and function calling — the model’s continuation asks to run a function; your code runs it and pastes the result back in as text.
- RAG — your code finds relevant documents and slips them into the context before the call.
- Agents — a loop of calls, where each turn the model’s continuation names the next action, your code executes it, and the result goes back in.
One mechanism — predict the continuation — different text. Once you see this, “building with AI” stops being mystical: it is deciding what text goes in and what happens to the text that comes out.
Why models make things up
Now the big question: if the model is this good, why does it confidently state things that are simply false — fake citations, invented functions, wrong dates?
Because it predicts plausible text, not true text. “Plausible” and “true” overlap most of the time — that’s why LLMs are so useful — but they are not the same thing. When the model hits a spot where it doesn’t really know the answer, it doesn’t stop and say nothing. It continues with the most statistically natural-sounding continuation. That failure mode has a name: hallucination.
This is not a bug that will be patched next Tuesday. It is a direct consequence of how the mechanism works. Everything practical in this roadmap — grounding models in your data with RAG, giving them tools to check facts, forcing structured output, running evaluations — exists to work around this one property.
The engineer's rule
Never treat a model’s output as a source of truth. Treat it as a very good draft that needs verification — and design your product so that verification is cheap or automatic.
One paragraph on AI vs AGI
Does next-token prediction add up to “real intelligence”? People argue about this constantly. The honest engineering answer: it doesn’t matter for your day-to-day work yet. What we have today is narrow AI — extremely capable tools that still need a human to define the goal, check the result and own the consequences. Whether and when we reach AGI — systems with human-level general ability — is a fascinating question, but it changes nothing about how you build a reliable feature this year.
You ask the same question twice and get two different answers. What happened?
You close the chat and open a new one. Why does the model no longer know your name?
Why does a model sometimes state a fake fact very confidently?
Questions?
Stuck or have questions about this roadmap? Join the Telegram channel — happy to discuss.