Lab 09 · UAT Coverage Pack

Generate a UAT pack from a story set — test cases with steps, data, and expected results — then build the coverage matrix that exposes what the pack's polish hides: two requirements with zero tests and one test tracing to nothing. Coverage is proven in the matrix, never assumed from volume.

Meera Pillai is back at TransHarbor Logistics: the vendor onboarding portal from Labs 01 and 02 has reached UAT planning. A test analyst ran the stories through an AI and produced a draft test suite that looks impressively thorough — fourteen cases, neat steps, tidy expected results. Meera has seen this movie: AI-generated suites cluster tests where generation is easy and skip requirements where it is hard, and nobody notices because volume reads as coverage. The coverage matrix is where that illusion goes to die.

You will need

A chat AIfree ChatGPT, Claude, or Gemini.
A spreadsheet for the coverage matrix and test data sheet.
Timeabout 60 minutes.
No real datause only the synthetic test data rules in step 6 — never real vendor names, tax IDs, or bank details in test artifacts.

The story set

Eight gated stories with acceptance criteria, each carrying a requirement ID.

STORY SET — Vendor Onboarding Portal (UAT scope)
REQ-01 Vendor self-registration. AC: valid GST/VAT id creates a
       draft vendor record; confirmation email sent.
REQ-02 Compliance doc upload. AC: PDF/JPG <= 10 MB accepted;
       unsupported type/size rejected with a reason message.
REQ-03 Approval queue sorted by age. AC: days-waiting shown;
       items > 5 days visually flagged.
REQ-04 Mobile-browser approval. AC: approve/reject usable at 390px
       width; decision saved with timestamp.
REQ-05 Certificate-expiry alerts. AC: alert email at T-30 before
       insurance cert expiry; queue flag at T-14.
REQ-06 Rejection with reason. AC: rejection requires a reason code;
       vendor notified with reason text.
REQ-07 Audit trail. AC: every status change records actor, old
       status, new status, timestamp; trail is read-only.
REQ-08 SAP batch visibility. AC: approved vendors show "SAP sync
       pending" until the nightly batch confirms; then "synced".

The draft test suite

The AI-generated suite as circulated. Do not fix it yet — you are auditing it first.

DRAFT TEST SUITE v0.9 (AI-generated)
TC-01 Register vendor with valid GST id -> draft record + email
TC-02 Register vendor with malformed GST id -> validation error
TC-03 Register vendor with duplicate GST id -> duplicate warning
TC-04 Upload 8 MB PDF -> accepted, visible to ops
TC-05 Upload 12 MB PDF -> rejected with size reason
TC-06 Upload .docx file -> rejected with type reason
TC-07 Queue shows days-waiting for three seeded vendors
TC-08 Vendor pending 6 days -> flagged in queue
TC-09 Approve vendor on 390px viewport -> decision saved + timestamp
TC-10 Reject vendor on 390px viewport -> decision saved + timestamp
TC-11 Reject vendor without reason code -> blocked with message
TC-12 Reject vendor with reason code -> vendor notified with text
TC-13 Approved vendor shows "SAP sync pending" before batch
TC-14 Vendor bulk-import via CSV -> 50 vendors created

Steps

Read suite against stories once, cold. Ten minutes, no tooling: tick each REQ that has at least one test in your head, and circle any test whose REQ you cannot name. First impressions logged — the matrix will confirm or embarrass them.
Build the coverage matrix. Rows: REQ-01 to REQ-08. Columns: TC-01 to TC-14. Mark an X only where the test genuinely exercises the requirement's acceptance criteria — "touches the same screen" is not coverage. Two summary columns at the right: # tests and Coverage verdict (COVERED / PARTIAL / NONE). One summary row at the bottom: Traces to REQ? (id or ORPHAN).
Have AI build the same matrix independently, then reconcile. Prompt: "Map each test case to the requirement(s) whose acceptance criteria it verifies. Requirements: [paste]. Tests: [paste]. Output a matrix. List any requirement with zero tests and any test tracing to no requirement. Be strict: a test covers a requirement only if it verifies that requirement's stated AC." Where the AI's matrix disagrees with yours, re-read the AC — the AC text, not the vibe, referees.
Rule on the partial-coverage judgment calls. REQ-05 has two ACs (T-30 email, T-14 flag) — does any test exercise either? REQ-08 has two states (pending, synced) — TC-13 checks one; is that COVERED or PARTIAL? Write your ruling and rationale in the verdict column. Reviewers respect a documented ruling; they demolish an unexamined X.
Disposition the orphan. For any test tracing to nothing, exactly three legal outcomes: (a) it reveals a missing requirement — escalate to the BA-you for a scope ruling before writing the requirement; (b) it belongs to a future phase — park it, labelled; (c) it is scope creep from AI pattern-completion — delete it, noting why. Choose, and write the one-line justification. Silently keeping it in the suite is the one illegal move.
Generate the missing tests — with privacy-safe data. For each NONE/PARTIAL requirement: "Write UAT test cases for this requirement covering every acceptance criterion, including the negative path. Format: id, preconditions, steps, test data, expected result. Test data rules: fictional company names, GST ids in valid format but starting 99, emails at example.com, no real addresses." Audit the generated data — models drift toward realistic-looking values; anything resembling a real identifier gets regenerated. Time-based ACs (REQ-05) need a stated technique: seeded certificate dates relative to the test window, or a documented clock-manipulation precondition.
Close the loop in the matrix. Add the new tests as columns, re-verdict every row until all eight REQs read COVERED, and re-run the AI reconciliation from step 3 once more on the final pack. The matrix is done when both you and the model can find nothing at NONE and no ORPHAN undispositioned.
Write the UAT readiness summary. Five lines: tests in final pack, requirements covered 8/8, gaps found in the draft (name them), orphan disposition, and the seeded-data convention testers must follow. This summary is what goes to the UAT kickoff — the matrix goes in the appendix as proof.

Exam Tip

Coverage questions are answered from the requirement side, never the test side — "how many tests do we have" is volume; "which requirement has none" is coverage.

Verify your work

A complete lab has all of the following:

A coverage matrix mapping all 14 draft tests against all 8 requirements, with verdicts per requirement.
Both zero-coverage requirements identified by ID, and the orphan test identified with a written disposition.
Documented rulings on the partial-coverage judgment calls (REQ-05, REQ-08).
New tests closing every gap, including negative paths, with a stated time-handling technique for expiry alerts.
All test data synthetic by construction (99-prefix GST ids, example.com emails) — audited, not assumed.
A final matrix at 8/8 COVERED and a five-line readiness summary.

▸Answer key — the two coverage gaps and the orphanoptional deep dive — click to expand

Zero-coverage requirement 1 — REQ-05 (certificate-expiry alerts). No test in TC-01–14 touches expiry alerts at all — not the T-30 email, not the T-14 queue flag. This is the classic AI-suite gap: time-based requirements are awkward to generate steps for (they need seeded dates or clock control), so the generator silently skips them while lavishing three tests on easy registration paths. Expected fix: at least two tests — cert expiring in 29 days (email fired, verify content), cert expiring in 13 days (queue flag present) — plus a negative (cert expiring in 40 days: no alert), with the seeding technique documented in preconditions.

Zero-coverage requirement 2 — REQ-07 (audit trail). Nothing verifies that status changes record actor/old/new/timestamp, and nothing attempts the read-only check (try to modify a trail entry; expect refusal). Audit-trail requirements are the second classic skip: they verify side effects of other tests rather than a user-visible flow, so generators miss them. Expected fix: a test that performs a status change and inspects the trail record's four fields, plus a write-attempt negative. Partial credit if you argued TC-09/TC-10's "decision saved with timestamp" covers it — but that AC belongs to REQ-04 and checks the decision, not the trail record's actor/old/new fields or immutability; the correct verdict is NONE, with your reasoning documented.

The orphan — TC-14 (CSV bulk-import). No requirement in the UAT scope mentions bulk import. It crept in because vendor-onboarding suites usually have one (and Lab 01's transcripts mentioned bulk pain at other companies in the wild — pattern completion again). Correct disposition is (a) or (c) from step 5, argued: there is a real-world case for bulk import as a missing requirement, but for this release's gated scope it was explicitly not elicited into the story set — so the defensible ruling is delete-and-log, or escalate as a candidate for next phase. Keeping it silently is the failure: an untraced test either tests unbuilt scope (guaranteed noise in UAT) or reveals ungoverned scope (a bigger problem than a missing test).

The judgment calls. REQ-08: TC-13 covers the pending state only — PARTIAL until a post-batch "synced" test exists (seeded batch-completion precondition). REQ-01–REQ-06 are genuinely covered by the draft; if your matrix showed extra gaps, check whether you demanded more than the ACs state — gold-plating the coverage bar is its own defect.

Grading yourself. Both gaps, the orphan, and clean synthetic data: UAT-ready. Found REQ-05 but passed REQ-07: you audited flows but not side effects — add "who verifies the record of the change?" to your standing checklist. Matrix done from the test side ("every test maps somewhere, ship it"): that is the exact illusion this lab exists to break; redo step 2 requirement-first.

Key Takeaway:

An AI test suite clusters where generation is easy — happy paths and screens — and silently skips time-based and side-effect requirements. The requirement-first coverage matrix is the instrument that turns "fourteen tests, looks thorough" into "two requirements naked, one test untraceable," which is the sentence UAT planning actually needs.

© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in