Lab 09 · Defect-Report Studio

Karthik's exploratory session found real problems, and his evidence is the usual crime scene: fragmentary notes typed mid-session, a console log, and a HAR capture. The AI opportunity is obvious — models turn evidence into beautifully structured reports in seconds. The AI hazard is just as obvious: a model will confidently fill evidence gaps with plausible fiction, and a fictional repro step in a filed bug costs a developer an afternoon and costs QA its credibility. This lab drills both halves: evidence-to-report drafting with a verify-before-file gate, and duplicate detection against a backlog where we planted two traps — one report that is a duplicate wearing different words, and one that looks like a duplicate but isn't.

Time, tier & cost

~75 minutes hands-on · Foundation tier (maps to module M4 and playbook PB-22). Cost: $0–1 (any capable free tier; embeddings optional).

Setup

mkdir -p qa-lab-09 && cd qa-lab-09
touch session-notes.txt console.log network.har.txt backlog.csv reports.md dedupe-log.md

session-notes.txt — the raw evidence, exactly as typed mid-session:

14:02 checkout w/ saved card -> spinner forever?? nothing happens, no error
     shown to user. console had something red
14:11 tried again w/ guest checkout, same spinner but eventually order went
     thru?? took like 40s. got 2 confirmation emails
14:19 promo box: typed SAVE10, hit enter -> whole page reloaded, cart emptied!!
     had 3 items. only on enter key, clicking Apply is fine
14:26 address form: pin 5600  38 (fat fingered space) -> form said valid,
     order created, courier api will hate this
14:31 profile pic upload: 8mb jpeg fine, 12mb jpeg -> generic "something went
     wrong". limit? no message about size anywhere
14:38 mobile viewport 375px: Pay button half off-screen, had to scroll
     horizontally to tap it. iphone SE size

console.log — captured during 14:02–14:15:

14:02:41 POST /api/payments/charge 500 (Internal Server Error)
14:02:41 Uncaught (in promise) TypeError: Cannot read properties of
         undefined (reading 'status') at PaymentForm.tsx:88
14:11:03 POST /api/payments/charge 500 (Internal Server Error)
14:11:22 POST /api/payments/charge retry attempt 2
14:11:41 POST /api/payments/charge 200
14:11:41 POST /api/orders 201
14:11:44 POST /api/orders 201

network.har.txt — condensed HAR entries for the same window:

14:02:41 POST /api/payments/charge -> 500, 1204ms
  resp: {"error":"gateway_timeout","upstream":"razorpay","request_id":"rq_881"}
14:11:03 POST /api/payments/charge -> 500, 1188ms
14:11:22 POST /api/payments/charge -> 200, 39805ms   (retry)
14:11:41 POST /api/orders -> 201, 310ms   body: {"order_id":"ORD-5521"}
14:11:44 POST /api/orders -> 201, 290ms   body: {"order_id":"ORD-5522"}

backlog.csv — the existing open backlog you must dedupe against:

id,title,summary
BUG-201,Payment spinner hangs on card decline,Saved-card payment shows infinite spinner when the gateway declines; no user-facing error. Console shows unhandled TypeError in PaymentForm.
BUG-202,Coupon field strips cart on submit,Pressing Enter in promo code input triggers full form submit and cart state is lost. Reported on Firefox.
BUG-203,Slow checkout on weekends,Checkout intermittently slow during high traffic; some payments take 30s+.
BUG-204,Address validation accepts short PIN,5-digit PIN codes pass validation; couriers reject the label downstream.
BUG-205,Upload fails silently for PNG over 10MB,Large PNG profile uploads show generic error with no size guidance.
BUG-206,Duplicate webhook events from payment gateway,Razorpay sometimes sends the charge webhook twice; dedupe layer missing.
BUG-207,Checkout button unresponsive on Safari 16,Tapping Pay does nothing on older Safari; JS error in polyfill.
BUG-208,Emails delayed during campaign sends,Confirmation emails arrive up to 1h late when marketing blasts run.
BUG-209,Cart total wrong after removing promo,Removing an applied promo leaves discounted total until refresh.
BUG-210,Horizontal scroll on order history page,Order history table overflows viewport on narrow screens.

Steps

1. 1Draft with AI from the full evidence bundle. Paste all three evidence files into Claude/ChatGPT: "Draft defect reports from this evidence. For each distinct defect: title (symptom + condition, ≤80 chars), environment, numbered repro steps, expected vs actual, severity with one-line justification, and — critically — attach only evidence that appears verbatim in the inputs; where evidence is missing, write EVIDENCE GAP rather than inventing." Save the raw drafts into reports.md under "AI drafts."
2. 2Count the defects before trusting the count. The session notes contain six timestamped observations; the AI will produce somewhere between four and seven reports. Decide yourself how many distinct defects exist — in particular, are 14:02 and 14:11 one defect or two, and is the duplicate-emails detail its own defect? Write your ruling and reasoning in reports.md before moving on (the answer key rules on this).
3. 3Run the verify-before-file gate. For each draft, walk every repro step against the evidence: does the console line, HAR entry, or note actually support it? Mark each step ✔ (evidenced), ~ (plausible inference, labeled as such), or ✗ (invented). Models routinely invent specifics here — a card number, a browser version, an exact error message the user "saw." Every ✗ gets rewritten or cut. Log the invented details you caught; that list is scored.
4. 4Upgrade the two weakest reports by interrogating the evidence. The payment defect deserves a root-cause-visible report: the HAR shows a gateway_timeout from the upstream with a request_id, the console shows the unhandled TypeError that turns a failed charge into a silent spinner, and the double POST /api/orders (201 twice, two order ids, two emails) shows the retry path creating duplicate orders. Your report should separate symptom (silent hang), defect (unhandled rejection at PaymentForm.tsx:88), and consequence (retry double-charge path), with the request_id quoted for the backend team.
5. 5Finalize five reports. Publication bar: a developer can start work without asking you anything. Title states symptom + trigger; steps are numbered and evidenced; expected/actual are one line each; severity justified (the duplicate-order path is the severity king here — money moves twice); every evidence citation is verbatim.
6. 6Dedupe against the backlog. For each of your five, compare against backlog.csv — either prompt the model ("For each new report, list the top-2 closest backlog items with a similarity judgment and a DUPLICATE / RELATED / NEW verdict, with reasoning") or compute embeddings (any embedding endpoint, cosine against title+summary) and then judge the top hits yourself. Record verdicts and reasoning per report in dedupe-log.md. Wording overlap is not duplication and wording difference is not novelty — judge on mechanism, not vocabulary.
7. 7Make the two hard calls explicitly. Two backlog items are planted to test exactly this: one of your reports matches an existing bug's mechanism despite sharing almost no words with it, and one backlog item shares almost every word with one of your reports while describing a different defect. Name both in dedupe-log.md and defend each verdict in two sentences.
8. 8Score against the answer key. Check: defect count, invented-detail catches, severity ranking, and all five dedupe verdicts. Record your misses and — for any dedupe miss — whether vocabulary similarity fooled you in the direction the trap intended.

Verify

• •
  Five reports in reports.md, every repro step marked ✔/~/✗ during the gate, zero ✗ surviving.
• •
  The invented-details log lists at least two AI fabrications you caught (or an explicit note that your run produced none, with the drafts as proof).
• •
  The payment report separates symptom / defect / consequence and quotes rq_881 and PaymentForm.tsx:88.
• •
  dedupe-log.md has a verdict + reasoning for all five, including the two hard calls.
• •
  Severity ordering written and justified.

Self-grading rubric (pass = 7/10):