Lab 11 · First LLM-Feature Test Suite

Sara is an SDET taking her first step into AI evaluation engineering, and the first wall every tester hits is the same: the feature returns a different string every run, and the entire assertion toolbox assumes it won't. The wrong responses to that wall are giving up on testing ("it's non-deterministic, we'll eyeball it") or over-reaching for an LLM judge on every assertion (slow, costly, and itself non-deterministic). The right response is the layered hierarchy — pin everything that is deterministic, measure meaning where wording varies, and spend judge calls only on qualities nothing cheaper can see — with pass criteria defined statistically over N runs, not existentially over one.

Time, tier & cost

~90 minutes hands-on · Core tier (maps to module M6 and playbooks PB-34/PB-35). Cost: $0 with Ollama local; ≤$2 via a hosted API.

Setup

mkdir -p qa-lab-11 && cd qa-lab-11
python3 -m venv .venv && source .venv/bin/activate
pip install pytest requests sentence-transformers
ollama pull llama3.1:8b        # or set an API key for a hosted model
npm install -g promptfoo       # optional path for step 7

feature.py — the feature under test (a support-ticket summarizer):

import requests, json

PROMPT = """Summarize this support ticket for the engineering triage board.
Rules: max 40 words; must state the affected feature; must state customer
impact; must not include the customer's name or email; end with a severity
guess formatted exactly as [SEV:high|medium|low].

Ticket:
{ticket}"""

def summarize(ticket: str, temperature: float = 0.7) -> str:
    r = requests.post("http://localhost:11434/api/generate", json={
        "model": "llama3.1:8b",
        "prompt": PROMPT.format(ticket=ticket),
        "stream": False,
        "options": {"temperature": temperature},
    })
    return r.json()["response"].strip()

tickets.py — the golden inputs with expected properties (not expected strings):

TICKETS = [
  {
    "id": "T1",
    "text": ("From: meera.k@bigcorp.example — Checkout fails for our whole "
             "team since 9am. Card page throws error after OTP. We process "
             "300 orders/day and are fully blocked. Please escalate."),
    "must_mention": ["checkout"],
    "must_not_mention": ["meera", "bigcorp", "meera.k@bigcorp.example"],
    "expected_sev": "high",
    "reference": ("Checkout is failing after OTP entry, fully blocking a "
                  "customer who processes 300 orders daily. [SEV:high]"),
  },
  {
    "id": "T2",
    "text": ("From: dev@tinyshop.example — The CSV export of last month's "
             "invoices has the date column in US format instead of ISO. Not "
             "urgent, we reformat manually, but please fix eventually."),
    "must_mention": ["export"],
    "must_not_mention": ["tinyshop", "dev@tinyshop.example"],
    "expected_sev": "low",
    "reference": ("Invoice CSV export uses US date format instead of ISO; "
                  "customer has a manual workaround. [SEV:low]"),
  },
]

Steps

1. 1Prove the exact-match failure with data, not doctrine. Write explore.py: call summarize(TICKETS[0]["text"]) five times at temperature 0.7, print all five outputs and whether any two are byte-identical. Then run five more at temperature 0. Record both results in rationale.md. Expected: zero identical pairs at 0.7; at temperature 0 you may see identical or near-identical outputs — and you must not rely on it, because provider-side changes, hardware nondeterminism, and model updates break byte-stability even at temp 0. One paragraph: why assert output == expected has a false-failure rate near 100% here.
2. 2Name the flakiness-vs-variance distinction. In rationale.md, write the two definitions in your own words: flakiness is the same system giving different pass/fail results for reasons unrelated to quality (bad test); variance is the system legitimately producing different outputs of equivalent quality (property of the feature — the test must assert over the distribution, not the instance). Every design choice below hangs off this distinction.
3. 3Build Layer 1 — deterministic checks. In test_layer1.py, write pytest tests that must hold on every single output with zero tolerance: word count ≤ 40 (the spec says so); the severity tag present and exactly matching \[SEV:(high|medium|low)\]$; none of the must_not_mention strings present case-insensitively (the PII rule); non-empty output. These are cheap, instant, and non-negotiable — a PII leak in 1 of 20 runs is a failed feature, not variance. Run each test over 5 samples per ticket with the rule: any Layer-1 failure in any sample fails the test.
4. 4Build Layer 2 — semantic similarity. In test_layer2.py, embed each output and its reference with sentence-transformers (all-MiniLM-L6-v2) and assert cosine similarity above a threshold. Calibrate the threshold empirically, not by folklore: compute similarity for 5 good outputs (from step 1) and for two deliberately bad ones you write by hand — a summary of the wrong ticket, and a fluent-but-vacuous "A customer reported an issue. [SEV:medium]". Set the threshold to separate the clusters (typically lands around 0.55–0.70 for this model; yours may differ — the method is the skill). Record the calibration table in rationale.md.
5. 5Build Layer 3 — judge, narrowly. Only two qualities here genuinely need a judge: "states customer impact" and "severity guess is appropriate to the ticket." In test_layer3.py, call the model as judge with a binary rubric per quality ("Answer PASS or FAIL only: does this summary state the customer impact? Summary: ..."), and assert on the parsed verdict. Keep the judge prompt output-constrained and log raw judge responses to a file — you will thank yourself when a verdict looks wrong. Note in rationale.md why coherence/fluency did not get a judge: Layer 2 already correlates with it, and judges cost latency and money per assertion.
6. 6Wrap it in statistical gates. Refactor into test_suite.py with an N-run harness: for each ticket, generate N=5 outputs once (a session-scoped fixture — do not re-call the model per assertion), then apply: Layer 1 must pass 5/5 (zero-tolerance), Layer 2 must pass ≥4/5, Layer 3 must pass ≥4/5. Justify each threshold in rationale.md: which failures indicate a broken feature at any frequency, and which indicate acceptable tail variance. Run the full suite three times; all three runs must be green — a gate that flakes across suite-runs has its thresholds set on the noise floor.
7. 7Port the suite shape to Promptfoo (optional but recommended). Express the same design declaratively in promptfooconfig.yaml: the prompt, the Ollama provider, per-ticket tests with contains/regex/not-contains asserts (Layer 1), a similar assert with your calibrated threshold (Layer 2), and an llm-rubric assert (Layer 3), with repeat: 5. Run promptfoo eval and confirm it agrees with pytest. Knowing both idioms — pytest-native and config-native — is the employable skill.
8. 8Finish the rationale. Complete rationale.md: the variance evidence, the layer table (assertion → layer → cost → tolerance → why), threshold calibration data, and the one-line summary of the hierarchy rule: cheapest sufficient layer first; zero tolerance only where any failure is a defect; statistics everywhere wording varies.

Verify

• •
  rationale.md contains the 5+5 run evidence with the identical-pair count, and the flakiness-vs-variance definitions.
• •
  Layer 1 tests: word cap, severity regex, PII absence, all at 5/5 zero-tolerance.
• •
  Layer 2 threshold set from a recorded calibration table including the two hand-written bad outputs.
• •
  Layer 3 judges exactly two qualities, binary rubric, raw responses logged.
• •
  Full suite green three consecutive runs; model called once per ticket-batch via fixture, not per assertion.
• •
  Layer table in the rationale maps every assertion to layer, cost, and tolerance.

Self-grading rubric (pass = 7/10):