The day your product ships an LLM feature, expect(output).toBe(x) dies — and with it, most of what your test suite assumed about the world. This module rebuilds the foundations: statistical pass criteria for non-deterministic output, the layered assertion hierarchy, golden datasets as the new regression pack, and prompt regression gates in CI.
This is where the genuinely new discipline begins — and where the career premium lives. Job boards now carry titles that did not exist three years ago: AI QA engineer, AI evaluation engineer, LLM test engineer, with postings naming "LLM tooling in CI/CD" and "prompt regression" verbatim, and AI-skill salary premiums running 28–43% (Lightcast). The skills are not exotic: they are QA instincts — regression packs, assertions, CI gates — re-derived for systems that answer differently every time you ask. An SDET who internalizes this module is not learning a new job; they are extending the one they have into the decade where every product has an AI feature.
Sara — SDET, five years, at a US SaaS company — is assigned quality for the new support copilot: an LLM feature that summarizes ticket threads for agents. She does the obvious thing: capture a good summary, assert equality. expect(summary).toBe(golden). It fails on the second run. Same input, same model, different words — same meaning.
The team's reflex diagnosis: flaky test. The retry culture from Module 4 kicks in — run it until it passes. It passes on attempt three. Everyone moves on. Sara stares at the green check with a bad feeling: the test now passes when the output happens to match, and tells you nothing when it doesn't.
A prompt "improvement" ships — someone tightened the wording to reduce token cost. Nothing failed, because nothing could fail: the equality asserts had been retried into meaninglessness. In production, summaries of refund-request threads quietly stop mentioning the refund amount. Support agents start missing commitments made to customers. The bug is discovered by an angry escalation, eleven days after the prompt change.
Sara rebuilds on eval foundations: a golden dataset of 60 real ticket threads with labeled expectations; layered assertions — schema and must-mention checks first, semantic similarity second, an LLM judge only for tone; every check run 5 times with statistical thresholds; the whole thing wired into CI so a prompt change that drops refund-amount mentions blocks the PR that caused it.
The next prompt tweak fails the gate in CI with a readable diff: "refund_amount mention-rate fell 100% → 40% on refund-thread slice." Fix, rerun, merge. Eleven days of production exposure becomes eleven minutes in a pull request.
Everything in Sara's rebuild is a concept this module teaches, and none of it required a data-science degree — it required recognizing that non-determinism is a property to be measured, not a flake to be retried.
An LLM generates text by sampling from a probability distribution over next tokens. At temperature 0 the sampling is greedy and mostly stable — but not contractually deterministic across infrastructure, batching and model updates. At production temperatures, the same input legitimately yields different phrasings, orderings and word choices on every call. This forces the distinction that Module 4's triage instincts must now absorb:
Treating variance as flakiness leads exactly where it led Sara's team: retry-until-green, which converts your suite into a coin-flip detector that passes when the coin lands right. The correct response to variance is statistical pass criteria — make the sample size and the threshold explicit:
Given a scenario where an LLM test "fails sometimes," the scored move is to classify before fixing: infrastructure flake (fix the test), legitimate variance (replace exact-match with a statistical criterion), or genuine quality instability (file it — a feature that is right only 60% of the time has a product problem, not a test problem).
If exact equality is dead, what asserts? A hierarchy, ordered by cost and applied bottom-up — the single most load-bearing design pattern in LLM testing:
Valid JSON against the Pydantic schema; length bounds; regex on required tokens; must-mention checks (the refund amount, the order ID); must-NOT checks (no PII, no competitor names, no apology spirals). Runs in microseconds, costs nothing, zero ambiguity.
Embedding similarity between output and reference answers a question Layer 1 cannot: "does this say approximately the right thing?" Tolerant of legitimate paraphrase, catches drift in substance. Costs an embedding call; thresholds need tuning per task against labeled examples.
A model grades the output against a rubric: professional tone, faithful to source, appropriately hedged. Most expressive, most expensive, and least trustworthy by default — a judge is itself an LLM with biases, which is why Module 7 requires calibration before any judge gates a release.
The discipline is economic and epistemic at once: never buy an expensive, fuzzy layer for a property a cheap, exact layer can check. Whether output is valid JSON is not a judge question. Whether the summary mentions the refund amount is a regex, not an embedding. Sara's suite runs forty deterministic checks, six semantic ones, and exactly one judged property — and the deterministic layer catches the majority of real regressions, for free.
| Layer | Checks | Cost per case | Failure ambiguity | Example |
|---|---|---|---|---|
| Deterministic | Schema, regex, must/must-not mention, length, PII-absence | ~0 | None — binary | Summary contains the ticket's refund amount |
| Semantic | Embedding similarity vs references, threshold | 1 embedding call | Low — tunable | Summary meaning ≈ labeled reference, ≥0.83 |
| LLM judge | Rubric-graded qualities | 1+ LLM calls | High — needs calibration | "Tone professional and non-committal: yes/no + reason" |
Every layer needs something to run against, and that something is the golden dataset: a versioned, labeled collection of real inputs with expectations — the direct descendant of the regression pack, rebuilt for AI features. Construction is curation, not generation:
The golden dataset is a living asset with a feedback loop — production failures flow in as new cases, stale cases get re-labeled when the product changes, and dataset diffs get reviewed like code diffs. A frozen dataset decays into testing last year's product; an ungoverned one silently redefines quality.
Prompts are code: they change behavior, they regress, and they deserve the same merge gate as everything else QA protects. Promptfoo is the declarative backbone — 350k+ developers, free and local-first — and the mental model is a test matrix: prompts × providers × dataset cases × assertions.
# promptfooconfig.yaml — the regression gate for the ticket summarizer
prompts:
- file://prompts/summarize_v3.txt
- file://prompts/summarize_v4_candidate.txt
providers:
- anthropic:claude-sonnet-4-5
tests:
- file://golden/refund_threads.yaml # per-case asserts ride with the data:
# assert:
# - type: is-json # Layer 1
# - type: icontains
# value: "{{expected_refund_amount}}"
# - type: similar # Layer 2
# value: "{{reference_summary}}"
# threshold: 0.83
Run locally while iterating; run in CI on every PR that touches a prompt, a model version or the dataset; fail the pipeline when pass rates drop against thresholds. The matrix answers the two questions that recur for the rest of your career: did the new prompt regress anything? (v3 vs v4 across the whole dataset, per-slice diffs in the output) and does the cheaper/newer model hold up? (same prompt, providers compared on identical cases — the evidence for every "can we switch models?" meeting).
For teams that live in Python and pytest, DeepEval provides the same layers with QA-native ergonomics — tests are just pytest, metrics (G-Eval rubrics, answer relevancy, faithfulness) are asserts, and output is JUnit XML that lands in the CI dashboards your org already reads:
def test_summary_keeps_refund_commitment(refund_case):
out = summarize(refund_case.thread)
assert refund_case.amount in out.text # Layer 1
assert_test(LLMTestCase(input=refund_case.thread, actual_output=out.text),
[AnswerRelevancyMetric(threshold=0.8)]) # Layer 2/3
The strategic point outranks the tools: the decision — no prompt change merges without the eval suite passing — is what saved Sara's team, and it will survive any individual framework. Tools churn (Promptfoo's acquisition by OpenAI is a live governance watch-item; DeepEval iterates monthly); the gate is forever.
Most LLM features ship structured output — JSON that downstream code parses — which makes this the most classically QA-shaped corner of the new discipline: it is contract testing with a probability distribution attached (UC-53).
The regression is visible in the diff, attributed to the slice, and stopped in the PR — not discovered in an escalation eleven days later.
Testing LLM features is QA's own instincts under a probability distribution: regression packs become golden datasets, asserts become layered hierarchies with the cheapest sufficient layer first, and the merge gate becomes a statistical pass criterion with N and threshold written down. Variance is not flakiness — measure it, gate it, and let the PR that regresses an edge case fail in CI instead of in production.
Take this into Lab 11 (first LLM-feature test suite) — where run-to-run variance is the seeded lesson — and Lab 12 (prompt regression gate in CI), where a provided prompt "improvement" secretly regresses an edge case and your gate must block the merge. Companion playbooks: pb-34 through pb-39.