Regression Testing for Prompts

Fix one case, break five

Prompts are code — they just do not look like it. And like code, every change is a change to everything the prompt touches. You rephrase one instruction to fix an over-eager refund bot, and quietly the assistant stops formatting dates correctly, because your rephrase removed a word an unrelated instruction leaned on. Nothing errors. Five other cases break, silently, in production.

With traditional code, tests catch this. With prompts, most teams have nothing — so the fix is to give prompts the same safety net: a golden dataset, an automatic run, and a gate that blocks bad changes.

In plain language: a regression is something that used to work and quietly stopped — not because you meant to break it, but because you changed something next to it. Nothing errors, nothing crashes, the feature just gets worse in a place nobody was looking.

Why you should care

Prompt changes are the cheapest changes you will ever make and the easiest to get wrong, because there is no compiler to tell you. A golden set of thirty cases is one day of work, and it converts “the bot got worse, does anyone know why?” into a blocked pull request with the failing case in the log. Nothing else in this track has a better effort-to-pain ratio.

Words you will see

Word	What it means here
Regression	Something that used to work and quietly stopped.
Golden dataset	Cases a human reviewed and froze: the exam before every release.
Baseline	The result of the last known-good run. The new run is compared against it.
Gate	A rule that blocks a change when the score drops.
CI (continuous integration)	A robot that runs your checks on every change and enforces the gate.
Diff	Not just the pass rate — the exact cases that changed verdict.
Drift	Slow, silent degradation over weeks: stale documents, a provider quietly updating a model.
Nightly run	The full suite running every night, catching what the fast gate misses.

A concrete example: Spanish fixed, dates broken

Users complain your bot answers Spanish questions with a refusal. You add one sentence to the prompt: “always attempt an answer before refusing.” Spanish is fixed. Two weeks later the support team notices that invoices now show dates as March 3, 2026 instead of 2026-03-03, because the new sentence pushed the formatting instruction further down and the model started treating it as optional. One fix, one new problem, zero errors, two weeks of silent damage. A golden set of thirty cases containing one date-formatting case catches this in the pull request — before the merge, not after the invoices.

The golden dataset

A golden dataset is a set of input-output pairs you have decided are correct — the exam your assistant must pass before every release. Not generated on the fly, not “whatever the model said last time” — reviewed and frozen by a human.

Where the cases come from:

Real production conversations — the most valuable source. Pull the twenty most common question types, plus every conversation that got a thumbs-down or ended in a support ticket. Every incident becomes a new test case; the dataset grows in exactly the places where reality hurts.
Edge cases by hand — empty input, prompt injection attempts, off-topic questions, hostile users. The model’s behavior on weird input must stay stable too.
One case per known bug — if it broke once, it is in the set forever.

Ten to fifty cases is enough to start. They should be cheap to run: a full golden-set pass costs a few cents and a few minutes. If writing the first ten feels hard, let an assistant draft them and then review every line yourself — drafting is cheap, judgment is the part you cannot delegate:

Draft your first golden set — paste into ChatGPT or your coding agent

I have a customer support assistant. Help me build a golden test set
of 13 cases I will run before every prompt change.

For each case give me:
1. A realistic user question (short, like a real chat message)
2. What a correct answer MUST contain (phrases or facts)
3. What it must NEVER contain (phrases that would be wrong or unsafe)
4. Why this case matters (one line)

Cover:
- The 5 most common question types
- 3 edge cases: empty input, an off-topic question, a hostile user
- 2 attempts to make the assistant invent a policy that does not exist
- 1 case in Spanish and 1 in German (we support both)
- 1 case that checks output format (dates must be YYYY-MM-DD)

Return it as a table. Do not invent our policies: where you need a fact,
write TODO-FILL-IN so I can put the real one in.

The workflow

Every prompt change — and every model change — triggers the same loop:

Change prompt → run golden set → compare → block or deploy
→→→→
1. Change
You edit the prompt, swap a model, or tweak temperature.
The change lives on a branch, like any code change. No direct edits to production prompts.
out› branch: fix-refund-tone
1/5

CI, in one concept

You do not need a sophisticated setup. The concept is a single sentence: prompt changes are code changes, so they go through the same pipeline — a commit triggers an automatic eval run, and the merge is blocked when the score drops. A nightly run against the full golden set catches the rest, including silent breakage from model provider updates.

Here is what that gate actually looks like in a repository. Three files: the cases, the check that refuses to let a regression through, and a stand-in for the assistant:

The gate: cases in, exit code out

1[
2{
3  #98c379">"id": "refund-window",
4  #98c379">"question": "Can I refund my annual plan?",
5  #98c379">"must_contain": ["14 days"],
6  #98c379">"must_not_contain": ["monthly"],
7  #98c379">"why": "policy fact, wrong answer costs money"
8},
9{
10  #98c379">"id": "date-format",
11  #98c379">"question": "Show me my last invoice date",
12  #98c379">"must_contain": ["2026-03-03"],
13  #98c379">"must_not_contain": ["March 3"],
14  #98c379">"why": "broke once, stays in the set forever"
15},
16{
17  #98c379">"id": "off-topic",
18  #98c379">"question": "What is the meaning of life?",
19  #98c379">"must_contain": [],
20  #98c379">"must_not_contain": ["refund"],
21  #98c379">"why": "must not turn every question into a sales pitch"
22}
23]

Three lines of that file are the whole idea: run every case, compare against the baseline case by case, exit non-zero if anything that used to pass now fails. Everything else is plumbing.

Check yourself

What is a regression?

You fix the Spanish answers and two weeks later the date format breaks. What would have caught it?

What makes a case "golden"?

The CI gate blocks your merge: 2 cases fixed, 3 broken. What is the right move?

Your takeaway

Four moves, and the order matters:

Collect ten cases from reality. Your most common questions, plus every conversation that got a thumbs-down or became a support ticket. Every incident becomes a case.
Freeze them as the baseline. Reviewed by a human, in a file in the repo. “Whatever the model said last time” is a snapshot, not a test.
Run the set on every change, automatically. A gate nobody can forget is the only gate that works.
Add one case per bug, forever. The dataset should grow exactly where reality hurt you.

The promise is small and specific: after this, “the bot got worse” arrives as a blocked pull request naming the case, not as a customer email three weeks later.