Talking to a Model: Tokens, Context and Sampling

Everything is tokens

Why do you care about any of this? Because tokens are the unit you are billed in, the unit your memory limit is measured in, and the reason a “clever” prompt sometimes costs ten times more than you expected. Fifteen minutes here turns mystery bills into arithmetic.

The model does not read letters or words. It reads tokens — chunks of text from a fixed vocabulary, typically 3-4 characters long in English. “artificial” might be one token, or two, depending on the tokenizer (the small program that splits your text into tokens before the model sees it). Code, non-English languages and rare symbols often tokenize less efficiently — meaning they cost more and fill context faster.

Three practical consequences:

• You are billed per token, both for what you send and what you receive.
• The context window — everything the model can “see” in one request, its working memory — is measured in tokens. Modern models range from about 8k (roughly this article) to a million-plus.
• The model remembers nothing between requests. What looks like memory is your app resending the whole conversation every time. The model sees your prompt as one flat stream: system instructions, user messages, tool results, its own previous answers — all stitched together into tokens. There is no magic memory beyond that window.

The context window is a budget

Think of the context window as a budget that every part of your app spends from. Here is a realistic agent interaction — press play and watch the budget fill up:

Two things engineers keep re-learning the hard way:

1. Every turn resends everything. A 10-turn conversation does not cost 10 turns of tokens — it costs a growing triangle of repeated history. This is why a “simple chatbot” can get expensive.
2. Long context is not free memory. Models get noticeably worse at using information buried in the middle of a huge context (“lost in the middle”). More context is not the same as better answers.

When the window fills up, apps do compaction: summarize the old turns and drop the details, keeping only what matters. It is the same thing you would do handing a long support thread to a colleague — you would not paste 200 messages, you would write “customer ordered twice, wants a refund for one”. (Tick “simulate context compaction” in the simulation above: the bulky tool result drops out while the pinned system prompt stays. That is the pattern: instructions must survive compaction, bulk must not.)

Sampling: how one token is chosen

Now the most misunderstood part. The model does not output text — it outputs a probability for every token in its vocabulary (its complete list of known chunks: tens of thousands of them). Something has to turn those probabilities into one actual choice. That is sampling — a weighted dice roll — and you control how the dice are loaded.

The headline parameter is temperature. Before picking a token, the model divides its raw scores by the temperature. High temperature flattens the distribution — unlikely tokens become more likely, like shuffling the dice so every face is nearly equal. Low temperature sharpens it — the most probable token becomes nearly certain. At temperature 0 the model always picks the single most likely token, so the same input gives the same output.

Play with it:

Try temperature at 0.0 several times: you always get “ Paris”. Now push it to 1.5 and sample a few times: “ London” and “ a” start showing up. Neither setting is “correct” — they are different tools:

• Low temperature (0.0-0.3) — extraction, classification, code, anything where you want the most probable answer, deterministically (same input, same output, every time).
• Mid (0.5-0.8) — general chat and writing: a bit of variety without nonsense.
• High (1.0+) — brainstorming, creative generation — when you want surprise.

The other knobs, briefly:

• top-p (nucleus sampling) — instead of a fixed number of candidates, keep the smallest set of tokens whose probabilities sum to p (e.g. 0.9) and sample from those. Think of it as “ignore the long tail of ridiculous options”. Often used alongside temperature; many teams just set temperature and leave top-p at its default.
• top-k — keep only the k most likely tokens, ignoring the rest. A blunter version of top-p.
• repetition penalty — a nudge that makes already-used tokens less likely, reducing loops in longer generations. Some APIs expose it, some do not.
• max tokens — a hard cap on how long the answer may be. Always set it: it is your safety net against a runaway bill.

An honesty check: temperature does not make the model “more creative” in any deep sense — it makes it noisier. Creativity you get from prompting; temperature just adjusts how far from the beaten path the sampler wanders.

One thing worth naming out loud: every knob on this page — temperature, top-p, max tokens — only changes the continuation of one call. The call itself never changes: text in, continuation out. Chat, agents, tools and RAG are that same call with different text going in. That idea is the spine of this whole course: The one call: how every AI product actually works.

Try it: copy this config

This is the config I reach for first in a new project. Copy it, change the model name, and adjust one number at a time — never two, or you will not know which one helped.

response = client.chat.completions.create(
  model="gpt-4o-mini",

  # 0.2 for facts and extraction, 0.7 for writing and chat
  temperature=0.2,

  # hard cap on the answer length - always set it
  max_tokens=500,

  # leave these alone unless you have a reason
  # top_p=0.9,
  # n=1,  # how many alternative answers to generate

  messages=[
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Your question here."},
  ],
)

Two settings that are easy to miss and worth knowing: n=1 means “give me one answer” — raise it to get several alternatives to choose from (and pay for each one). And seed, where supported, makes sampling reproducible: same seed, same input, same output — handy when you are debugging a prompt.

Streaming: why answers appear word by word

The model actually generates one token at a time, left to right — it cannot start the last word until it has written every word before it. APIs can send each token to you the moment it is produced — that is streaming. Nothing about the answer changes; only the delivery does. Compare it to a download: the file is the same size whether you watch the progress bar or not, but watching it feels much faster.

That is the whole trick: the user starts reading after the first few hundred milliseconds instead of waiting seconds for the finished block. For any interactive UI, streaming is not an optimization — it is table stakes.

Prompt caching: paying once for repeated context

Notice the simulation above: the system prompt and documents are resent with every request — and you pay full price every time. Many providers offer prompt caching: mark the stable prefix (the unchanging beginning of your prompt — system prompt, attached documents, tool definitions), and the provider reuses work it has already done instead of redoing it. Reading from cache typically costs a fraction of the normal input price and cuts time to first token.

The pattern to remember: stable stuff first, volatile stuff last. If your prompt interleaves changing and constant content, you defeat the cache. Put the big constant block at the start and the user’s actual message at the end.

Check yourself