Lab 01 · Test-Case Generation with a Review Gate

Feed an AI a requirements document that we have deliberately salted with ambiguities and one outright contradiction, let it generate a test suite, then run the review gate that separates professionals from prompt-pasters: score coverage, catch the hallucinated behaviors, and log every defect the AI invented or missed.

Meera is a QA engineer with six years on a payments team. Her manager wants "AI-generated test cases by Friday." The trap is not that AI writes bad test cases — it writes plausible ones, at speed, including confident cases for behavior that appears nowhere in the requirements. This lab makes you the gate. The requirements below contain planted defects; the AI will trip over some, paper over others, and invent a few of its own. Your job is to catch all of it and produce a corrected suite plus a caught-defects log.

Time, tier & cost

~75 minutes hands-on · Foundation tier (maps to module M1 and playbook PB-01). Cost: $0 with a free Claude/ChatGPT tier, under $1 via API.

Setup

Create a working folder and save the two files below exactly as written. Do not fix anything you notice in the requirements yet — the point is to see what the AI does with them.

mkdir -p qa-lab-01 && cd qa-lab-01
touch requirements.md coverage-rubric.md generated-cases.md caught-defects.md

requirements.md — the feature spec you will hand to the AI (defects planted, not marked):

FEATURE: GiftCard Redemption — release 4.2

FR-1  A customer may apply one gift card per order at checkout.
FR-2  Gift card codes are 12 characters, alphanumeric, case-insensitive.
FR-3  If the card balance exceeds the order total, the remainder stays
      on the card for future use.
FR-4  If the order total exceeds the card balance, the customer pays
      the difference with any supported payment method.
FR-5  Expired cards must be rejected with a clear error message.
FR-6  Redemption must feel fast for the customer.
FR-7  A card that has been reported lost is blocked within a reasonable
      time after the report.

ACCEPTANCE CRITERIA
AC-1  Applying a valid card with balance >= total sets amount due to 0.
AC-2  Applying a valid card with balance < total reduces amount due by
      the full card balance.
AC-3  A customer may apply up to two gift cards on a single order.
AC-4  Invalid code format shows error GC-400 without a server call.
AC-5  Expired card shows error GC-410.
AC-6  Blocked (lost/stolen) card shows error GC-423.
AC-7  Partial redemption records a ledger entry with the exact amount
      redeemed, in the card's issuing currency.

coverage-rubric.md — the gate you will score against:

Score each dimension 0-2 (0 = absent, 1 = partial, 2 = solid):
D1 Happy paths        — every FR/AC has at least one positive case
D2 Negative paths     — every stated error (GC-400/410/423) exercised
D3 Boundaries         — balance == total, code length 11/12/13, expiry
                        at the boundary instant
D4 Traceability       — every case cites the FR/AC it verifies
D5 No hallucinations  — zero cases testing behavior not in the spec
D6 Ambiguity flags    — vague requirements flagged as untestable
                        rather than silently "interpreted"
Max 12. Gate: a suite below 9 does not enter the test repo.

Steps

Generate the first draft blind. Paste requirements.md into Claude or ChatGPT with: "Act as a senior QA engineer. Generate a complete test-case suite for this feature. For each case give: ID, title, preconditions, steps, expected result, and the FR/AC it traces to. Output as a markdown table." Save the raw output to generated-cases.md untouched. Expect 15–30 cases in under a minute — that speed is exactly why the gate matters.
Write your own defect hypothesis first. Before reviewing the AI output, spend ten minutes reading requirements.md yourself and list in caught-defects.md everything you personally find ambiguous, contradictory, or untestable. This calibrates you against the machine — you should find at least three problems unaided.
Hunt the contradiction. Somewhere the requirements directly contradict themselves about the same behavior. Check what the AI did with it: most models silently pick one side and generate confident cases for it. Record in your log: which side did it pick, did it flag the conflict, and which test cases are now built on sand.
Hunt the hallucinations. Go case by case and ask of each expected result: "which line of the spec says this?" Typical inventions for this spec: a maximum card balance, a specific expiry-warning banner, a retry limit on invalid codes, a specific response-time number for FR-6, an email confirmation after redemption. Mark every case that tests unstated behavior with HALLUCINATED and the reason.
Score against the rubric. Fill in D1–D6 for the raw AI suite in coverage-rubric.md, with one line of evidence per score. Be strict on D6: an AI that turned "must feel fast" into "responds in under 2 seconds" did not resolve an ambiguity, it manufactured a requirement.
Interrogate the model. Feed the suite back with: "Review your own test suite against the original requirements. List every case that tests behavior not explicitly stated, and every requirement that is too ambiguous to test as written." Note in your log which planted defects it now admits, and which it still defends — a model doubling down on invented behavior is the key trust signal to record.
Produce the corrected suite. Rewrite generated-cases.md into final-suite.md: delete or rewrite hallucinated cases, add missing boundary cases (balance exactly equal to total; 11- and 13-character codes; expiry at the boundary instant with timezone called out as an open question), and convert every untestable requirement into an explicit OPEN QUESTION for the product owner entry rather than a test case.
Rescore and close the log. Score final-suite.md against the rubric — it should now clear the gate at 9+. Finish caught-defects.md with three columns: defect, who caught it (you in step 2, the review in steps 3–5, or the model's self-review in step 6), and how it would have leaked to production if the raw suite had shipped.

Warning

The failure mode this lab exists to prevent is not "AI wrote a wrong test case." It is a wrong test case passing review because it looked professional, then a real defect shipping because the suite verified invented behavior instead of specified behavior.

Verify

caught-defects.md lists the contradiction, at least three ambiguities, and at least two hallucinated cases — each with the exact spec line (or absence of one) as evidence.
Every hallucinated case in the raw suite is marked, and none survive into final-suite.md.
final-suite.md contains boundary cases for balance-equals-total and code length 11/12/13, and every case traces to an FR/AC.
Ambiguous requirements appear as open questions, not as tests with invented thresholds.
Both rubric scores are recorded, with the raw suite honestly below the gate if it deserved it.

Self-grading rubric (pass = 7/10):

Dimension	0	1	2
Contradiction caught	Missed	Noticed, cases not traced	Caught + affected cases listed
Ambiguities flagged	0–1 found	2 found	3+ found, all as open questions
Hallucinations purged	Some survive	All caught, weak evidence	All caught with spec-line evidence
Boundary coverage	Absent	Partial	Equal-total + length + expiry instant
Honest scoring	Rubric skipped	Scored leniently	Strict scores with evidence lines

▸Answer key — every planted defect (read only after step 8)optional deep dive — click to expand

The contradiction. FR-1 says "one gift card per order"; AC-3 says "up to two gift cards on a single order." These describe the same behavior and cannot both be true. Expected finding: the AI usually generates cases for one side (most often AC-3, because acceptance criteria read as more specific) without flagging the conflict. Full credit requires naming both lines, stating which side the AI chose, and listing every generated case that depends on the unresolved choice. This is the highest-value catch in the lab — score yourself 0 on "Contradiction caught" if you only found it via this key.

Planted ambiguity 1 — FR-6 "must feel fast." Untestable as written. A correct review converts it to an open question ("what is the latency budget, measured where, at what percentile?"). A hallucination-prone model converts it into a concrete number (2 seconds is the classic invention). Either way, any test case asserting a specific time is a planted-defect catch.

Planted ambiguity 2 — FR-7 "within a reasonable time." Same class: no SLA is stated for how quickly a lost-card report propagates to a block. Cases asserting "immediately" or "within X minutes" are hallucinations. Also hides a real design question your open-questions list should surface: what happens to a redemption in flight when the block lands mid-checkout?

Planted ambiguity 3 — FR-5 vs AC-5 expiry boundary. "Expired cards must be rejected" gives no boundary instant: is a card expiring today valid until 23:59:59, and in whose timezone — the customer's, the server's, or the card issuer's? Any case that tests expiry without flagging the timezone/instant question gets partial credit only.

Planted trap — AC-7 "in the card's issuing currency." The spec never states that multi-currency exists anywhere else, so this quietly implies a currency-conversion path with zero supporting requirements. Strong reviews flag it as an open question (what happens when card currency differs from order currency — rate source, rounding rule?). AIs almost never notice this one; a human catching it is the differentiator.

Expected hallucination inventory (any two suffice for full marks; all have appeared in real runs): maximum card balance or top-up rules; retry lockout after N invalid codes; expiry-warning UI; confirmation email; refund-to-card behavior; specific latency numbers for FR-6; card-stacking order rules beyond the contradictory count.

Scoring the raw AI suite: a typical unedited suite scores 6–8 of 12 — solid D1/D2, weak D3, superficially good D4 (it cites ACs, including for invented behavior — traceability to a hallucination is still a hallucination), and 0–1 on D5/D6. If you scored the raw suite 10+ honestly, either your model had an unusually strong run or your scoring was lenient — recheck D5 and D6 against this key.

Key Takeaway:

AI compresses test-case writing from days to minutes, which moves the entire professional skill into the review gate — coverage judgment, contradiction-hunting, and refusing to let invented behavior masquerade as a requirement. The generation was never the job; the gate is.

© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in