LLM Observability

Logs are not enough

A traditional web app fails loudly: a 500 error, a stack trace, an alert. An LLM app fails quietly. The request returns 200, the user gets an answer — a wrong, made-up, or ten-times-too-expensive answer. Nothing crashes; quality just slips. That is why AI features need observability beyond normal logs.

Three questions you must be able to answer for every request:

• What exactly did the model see? The full prompt, system message, retrieved chunks, tool results. When output is bad, the input usually explains it.
• What did it cost and how long did it take? Tokens in, tokens out, which model, total latency — how long the user waited. Costs hide in retries and long contexts.
• Did it work? A user thumbs-down, a retry, an abandoned session — signals that something went wrong.

Words you will see

One request, step by step

Click through a request to a support assistant with a RAG lookup and a tool call, and see what to log at each step:

Each step gets its own log line or span, and every span carries the same trace ID. Put together, those spans are a trace.

In plain language: a trace is one request’s diary. Every step, in order, filed under a single number — like the tracking page for a parcel, except instead of “arrived at the depot” you get “found 3 documents”, “sent 1 240 tokens to gpt-4o-mini”, “called the refund API”, “answered in 2.4 s for $0.0031”. When a user complains, you do not guess: you open the trace and see the whole request — what was retrieved, what was prompted, what came back, what it cost.

Tracing, monitoring, cost watch

Tracing is the detailed record above: every request decomposed into steps — prompt, model, tools, tokens, cost, latency — stitched together by a trace ID. It is your debugging tool: one request at a time, in full detail. When quality drops, you look at traces from bad sessions and find the pattern: the retriever started returning irrelevant chunks, or a prompt edit doubled the token count.

Production monitoring is the aggregated view — all your traces at once, turned into charts: error rates, latency percentiles, thumbs-down share, output length anomalies. It is your smoke alarm. It is also the difference between a user telling you “your bot got worse” and you noticing it three days earlier on a chart.

Cost and latency monitoring deserves its own line on the dashboard because LLM economics are unforgiving: a prompt that quietly grew from 800 to 8 000 tokens multiplies your bill, and a model switch can double latency. Track cost per request, per feature, per user cohort.

The tools

LangSmith — from the LangChain team. Deep tracing with dataset runs and eval features built in. The most complete option if you are in the LangChain ecosystem; works standalone too, but its mental model is LangChain-shaped.

Langfuse — open source, self-hostable, model-agnostic. Tracing, cost tracking, prompt management, evals. The default pick if you want your telemetry data to stay yours and do not want a LangChain tie-in.

Helicone — sits as a proxy between your app and the model API. Logging is nearly free to add: change one URL. Great for cost analytics and caching; less deep on application-level tracing than Langfuse.

Arize AI — the heavyweight, ML-observability heritage: drift detection, embedding analysis, online evals at scale. Worth it for large teams treating LLMs as part of a broader ML platform; overkill for a first AI feature.

What to instrument on day one

You do not need all of it at once. In order of value per minute of work:

1. One trace ID per request, threaded through every step. Without it you have logs, not a trace.
2. The full prompt and the full output. The two things you will look for in every single incident.
3. Model name and version, token counts, cost, latency. Four numbers that turn “it feels slow and expensive” into an argument with a chart.
4. Retrieval details — which documents were used and their scores. Most quality bugs are retrieval bugs wearing a costume.
5. Feedback — thumbs up, thumbs down, escalations. This is what turns traces into evals.

Try it: instrument one request

You do not have to write the plumbing by hand. Paste this into your coding agent — it will wire tracing into one endpoint and show you where to look:

Add tracing to the LLM call in this project, one endpoint only.

WHAT I NEED LOGGED, for every single request:
1. A trace ID that is created at the start and attached to every step.
2. The final prompt we sent (full text) and the full output we got back.
3. The model name and version.
4. Input tokens, output tokens, cost in cents, latency in milliseconds.
5. Any tool calls: name, arguments, raw response.

RULES:
- Do not log API keys, passwords, or anything that looks like personal data.
Redact them before writing.
- Explain in plain language where I can see the traces when I run this
locally, and give me one command to open them.
- Keep the change as small as possible: one endpoint, one file if you can.
- After you finish, tell me in two sentences what is logged and what is not.

Check yourself

Your takeaway

Four moves, in this order:

1. Pick one tool and instrument one request today. Not the whole app — one endpoint. You want to see a single trace on a screen before lunch.
2. Make sure the trace ID threads through every step. No ID, no trace, no debugging.
3. Put cost and latency on a dashboard from day one. These are the two numbers that quietly double.
4. Connect feedback to traces. A thumbs-down with a trace ID is the seed of your first golden dataset.

The habit: when something looks wrong, open the trace before you change the prompt. Guessing is more expensive than looking.