LLM Evaluations

“It seems fine” is not a QA strategy

You changed one word in the system prompt, clicked through three chats, everything looked fine, you shipped it. Two days later the assistant started refusing questions it answered perfectly before. Nobody noticed because nobody ran the other fifty cases.

This is the hole evaluations fill.

In plain language, an eval is a test for software that never answers the same way twice. With ordinary code you can say “this function must return 42”. With an LLM you cannot: it rewrites the wording every time. So you test a property instead — the answer must mention the 14-day window, must never promise a monthly refund, must be valid JSON. One case, one check, one point. Run them all and you get a score you can compare between releases. You do not need a huge suite on day one — you need ten honest test cases and the habit of running them.

Words you will see

Three kinds of checks

Deterministic — the old world still applies. Deterministic means: same input, same verdict, every single time, no judgment involved. If the output must be valid JSON, check it with a parser. If the answer must contain the tracking number, check substring presence. If a banned phrase must never appear, grep for it. These checks are free, instant and perfectly reliable — always start here.

Model-based (LLM-as-judge) — for things no regex can catch: “is this answer grounded in the provided context?”, “is the tone polite?”. You ask a strong model to grade the output against a rubric, ideally returning a score with a reason. It works surprisingly well, with two pitfalls: the judge is biased toward longer, more confident answers, and judges its own outputs generously. Never judge with the same model that produced the answer; give the judge explicit criteria, not “rate 1-10, vibes”.

Human evals — the ground truth, and the most expensive. Reserve them for the cases that matter most: a weekly review of a random sample of production conversations, plus every eval the model-judge flagged as borderline. Ten minutes of reading real transcripts beats an hour of speculation about quality.

Metrics worth tracking per release: pass rate, cost per case, latency, and the disagreement rate between judge and human — if the judge disagrees with people often, fix the judge before trusting it.

Which one should I use?

Start on the left, move right only when you have to.

The judge needs a rubric

An LLM judge given “rate this 1-10” will invent its own criteria and then change them tomorrow. Give it a rubric instead — a written list of what counts as good — and make it return a reason, so you can audit it later. Paste this into your agent or chat to get a first working judge:

You are grading one answer from a customer support assistant. Be strict and consistent.

QUESTION: [the user question]
CONTEXT WE GAVE THE ASSISTANT: [the retrieved documents, or "none"]
ANSWER TO GRADE: [the assistant's answer]

Grade each criterion as pass or fail:
1. GROUNDED: is every factual claim supported by the context, or does it
 invent a policy, price or date? Inventing anything is an automatic fail.
2. ANSWERED: does it actually address the question asked?
3. FORMAT: does it follow the required format (dates as YYYY-MM-DD, no
 markdown headings)?
4. TONE: polite, no apologies for existing, no over-promising.

Then reply with exactly this shape:
VERDICT: pass OR fail
FAILED_CRITERIA: [list the numbers that failed, or "none"]
REASON: [one sentence, quoting the exact words that made you decide]

Never give a score out of ten. Never guess a fact that is not in the context.

A minimal harness

Here is a complete eval loop — test cases, deterministic checks, a mocked model call, pass and fail counting. Read the comments: they are the whole design. The shape is exactly what you would run in production, minus the API key:

1# A minimal eval harness: cases, checks, a score.
2from mocks import assistant   # the app under test — swap for a real API call
3 
4# 1. THE CASES. Each case is one question plus what a good answer must do.
5#    expected_contains: phrases that MUST appear in the answer
6#    forbidden:         phrases that must NEVER appear
7#    Start with 10 real questions. Grow from production, not imagination.
8CASES = [
9  {
10      #98c379">"question": "Can I refund my annual plan?",
11      #98c379">"expected_contains": ["14 days"],
12      #98c379">"forbidden": ["monthly"],
13  },
14  {
15      #98c379">"question": "Reset my password",
16      #98c379">"expected_contains": ["login page"],
17      #98c379">"forbidden": [],
18  },
19  {
20      #98c379">"question": "What is the meaning of life?",
21      #98c379">"expected_contains": [],
22      #98c379">"forbidden": ["refund"],
23  },
24]
25 
26# 2. THE CHECK. One case in, True or False out.
27#    Deterministic: the same answer always gets the same verdict.
28#    Swap this function for an LLM-judge call when substrings run out —
29#    the loop below does not change.
30def check(case, answer):
31  has_expected = all(s in answer for s in case[#98c379">"expected_contains"])
32  has_forbidden = any(s in answer for s in case[#98c379">"forbidden"])
33  return has_expected and not has_forbidden
34 
35# 3. THE LOOP. Run every case, count passes, remember the failures.
36def run_evals():
37  passed, failed = 0, []
38  for case in CASES:
39      answer = assistant(case[#98c379">"question"])
40      if check(case, answer):
41          passed += 1
42      else:
43          failed.append(case[#98c379">"question"])
44 
45  # 4. THE SCORE. One number you can compare between two versions.
46  print(#98c379">"passed: " + str(passed) + "/" + str(len(CASES)))
47  for q in failed:
48      print(#98c379">"FAIL: " + q)      # the failing question, not just the count
49 
50  return passed == len(CASES)  # True = safe to ship
51 
52if __name__ == #98c379">"__main__":
53  run_evals()

Four parts, and that is the entire design: cases, a check, a loop, a score. The deterministic check catches the two most common regressions — the model stopped saying the right thing, or started saying something it must not. When you outgrow substrings, replace the check function with an LLM-judge call; the loop and the score reporting stay identical.

Check yourself

Your takeaway

Five moves, cheapest first:

1. Write ten cases today — real questions from real users, with what a good answer must contain and must never contain.
2. Check what you can check for free: valid JSON, required phrases present, forbidden phrases absent.
3. Add a judge only for what substrings cannot catch — and give it a rubric plus a reason, never a score out of ten.
4. Run the suite on every prompt or model change. Wire it into CI so a suite nobody remembers still runs forty times a week.
5. Track the baseline. A score you never recorded is a score you cannot compare against — and comparing is the entire point.

The rule of thumb: if you cannot say what would have to be true for your feature to be “good”, you do not have a feature yet. Write that down as ten cases and you do.