Lab 07 · Flaky-Test Triage Sprint

Triage a CI export of 40 failures that mixes real product bugs, async flakes, data collisions, order dependence, timezone traps, and infra noise. Use AI to cluster and classify, verify every cluster with a rerun strategy, quarantine correctly — and then grade your classifications against the labeled answer key.

It is 09:10 and Divya's nightly run is a wall of red: 40 failures across four suites. The old workflow is two engineers losing a morning to log-diving. The AI workflow is clustering the failures by signature in minutes — but AI classifications are hypotheses, and a hypothesis about flakiness is only worth what your rerun strategy can confirm. Misclassify a real bug as a flake and you quarantine a regression into production; misclassify a flake as a bug and you burn a developer's day on noise. This lab drills the full loop: cluster, classify, verify, quarantine, and ship the morning triage brief.

Time, tier & cost

~90 minutes hands-on · Core tier (maps to module M4 and playbooks PB-19/PB-20). Cost: $0 (the export is inline; any free AI tier works).

Setup

mkdir -p qa-lab-07 && cd qa-lab-07
touch failures.csv triage.md brief.md

failures.csv — the nightly export (run 6114, commit e9d21c7, 23:40–00:25 UTC). Save exactly:

id,test,suite,error_excerpt,duration_s,runner
1,checkout_submits_order,e2e,TimeoutError waiting for [data-test=confirm] 30s,31.2,r-04
2,order_api_create,api,500 InternalServerError: cannot read total of undefined,0.4,r-02
3,signup_new_user,api,409 duplicate key value violates unique constraint users_email_key,0.3,r-01
4,search_filters_apply,e2e,TimeoutError waiting for .results-grid 30s,30.9,r-04
5,order_api_update,api,500 InternalServerError: cannot read total of undefined,0.4,r-03
6,inventory_sync,integration,ECONNREFUSED db-primary:5432,0.1,r-07
7,report_daily_totals,unit,AssertionError: expected 2026-08-04 got 2026-08-05,0.1,r-02
8,login_remember_me,e2e,TimeoutError waiting for [data-test=dashboard] 30s,31.0,r-05
9,order_api_cancel,api,500 InternalServerError: cannot read total of undefined,0.4,r-02
10,profile_update_avatar,e2e,TimeoutError waiting for .avatar-preview 30s,30.8,r-04
11,signup_duplicate_check,api,409 duplicate key value violates unique constraint users_email_key,0.3,r-01
12,payments_refund_flow,integration,ECONNREFUSED db-primary:5432,0.1,r-07
13,cart_merge_on_login,e2e,TimeoutError waiting for [data-test=cart-count] 30s,31.1,r-05
14,order_api_list,api,500 InternalServerError: cannot read total of undefined,0.3,r-03
15,coupon_expiry_check,unit,AssertionError: expected expired=false got expired=true,0.1,r-02
16,email_verification,api,409 duplicate key value violates unique constraint users_email_key,0.2,r-01
17,wishlist_add_item,e2e,TimeoutError waiting for .wishlist-badge 30s,30.7,r-06
18,order_api_detail,api,500 InternalServerError: cannot read total of undefined,0.4,r-02
19,export_orders_csv,integration,ECONNREFUSED db-primary:5432,0.1,r-07
20,admin_bulk_edit,e2e,TimeoutError waiting for [data-test=save-toast] 30s,31.3,r-05
21,signup_password_rules,api,409 duplicate key value violates unique constraint users_email_key,0.3,r-01
22,order_api_totals,api,500 InternalServerError: cannot read total of undefined,0.4,r-03
23,notifications_badge,e2e,TimeoutError waiting for .notif-dot 30s,30.9,r-06
24,report_weekly_totals,unit,AssertionError: expected week 31 got week 32,0.1,r-02
25,user_prefs_theme,api,expected 200 got 404 Not Found /api/v1/prefs,0.2,r-01
26,shipping_rates_intl,integration,ECONNREFUSED db-primary:5432,0.1,r-07
27,checkout_guest_flow,e2e,TimeoutError waiting for [data-test=confirm] 30s,31.0,r-04
28,signup_welcome_email,api,409 duplicate key value violates unique constraint users_email_key,0.3,r-01
29,order_api_history,api,500 InternalServerError: cannot read total of undefined,0.4,r-02
30,product_gallery_zoom,e2e,TimeoutError waiting for .zoom-overlay 30s,30.8,r-06
31,session_timeout_redirect,e2e,TimeoutError waiting for [data-test=login-form] 30s,31.2,r-05
32,tax_calc_gst,unit,AssertionError: expected 118.00 got 100.00,0.1,r-03
33,user_prefs_language,api,expected 200 got 404 Not Found /api/v1/prefs,0.2,r-01
34,audit_log_write,integration,ECONNREFUSED db-primary:5432,0.1,r-07
35,signup_referral_code,api,409 duplicate key value violates unique constraint users_email_key,0.3,r-01
36,mobile_nav_drawer,e2e,TimeoutError waiting for .drawer-open 30s,30.9,r-06
37,order_api_search,api,500 InternalServerError: cannot read total of undefined,0.4,r-03
38,invoice_pdf_render,integration,ECONNREFUSED db-primary:5432,0.1,r-07
39,cart_total_rounding,unit,AssertionError: expected 99.99 got 100.00,0.1,r-03
40,loyalty_points_accrual,api,expected 200 got 404 Not Found /api/v1/prefs,0.2,r-01

Context you'd have in the room: commit e9d21c7 touched OrderService and renamed the prefs endpoint; the nightly crossed midnight UTC; runner r-07 had a maintenance window 23:55–00:10; the e2e suite runs against a shared staging environment; signup tests seed test.user@example.com.

Steps

Cluster with AI, eyes open. Paste the CSV and the context paragraph into Claude/ChatGPT: "Cluster these 40 CI failures by likely root cause. For each cluster: member ids, failure signature, most likely class (product-bug / test-bug / data / infra / environment-time), confidence, and the cheapest verification that would confirm or refute the classification." Save the raw response into triage.md under "AI hypothesis."
Challenge the boundaries. AI clustering tends to over-merge on surface text. Interrogate the two riskiest boundaries: "Are all 10 e2e timeouts really one cluster? What would distinguish an app-wide slowness flake from a broken shared dependency?" and "Are the five unit-test assertion failures one cluster or several?" Record how the clustering changes — the unit tests are not one cluster, and noticing that is scored.
Design the rerun strategy before rerunning anything. For each cluster, write in triage.md the verification you would run: rerun-same-commit ×N for suspected flakes (a real bug fails N/N, a flake fails intermittently), rerun-in-isolation for suspected order dependence, rerun-on-different-runner for infra, rerun-with-fresh-data for collisions, and no rerun but a code read for suspected product bugs (rerunning a real 500 forty times is how mornings die). Every cluster gets a named strategy and a predicted outcome.
Simulate the reruns against the key's behavior table. You don't have the real pipeline, so the answer key includes a "what happens when you rerun" table. For each cluster, apply your chosen strategy to that table and record whether your predicted outcome held. (Instructor-led cohorts: your facilitator plays the pipeline; self-paced: this is the one place you may open the key early — read only the rerun-behavior table, not the labels.)
Classify and quarantine. Give every cluster a final label and an action: file-bug-now (with owner), quarantine-and-ticket (flake), fix-test-data, infra-ticket, or fix-test (assertion/time handling). Quarantine means: tagged out of the merge-blocking run, ticketed with the flake signature, never silently deleted. List exactly which test ids you quarantine — quarantining a real product bug is the fail condition of this lab.
Write the two flake-class fix PRs. For the two flake classes you identified as test-side, write the fix as a described PR in triage.md: (a) the async/wait class — replace fixed 30s selector waits with web-first assertions plus one targeted app-side investigation if the shared staging env is genuinely slow; (b) the data-collision class — unique per-run emails (test.user+{runid}@example.com) or per-test fixtures with teardown. Include before/after code sketches for one representative test each.
Ship the morning brief. Write brief.md — ten lines max, the artifact a team lead reads at 09:15: N real bugs (ids, owner, severity), N flakes quarantined (classes, tickets), N infra (status), N test fixes in flight, and the one-line trend note. Every number must trace back to a cluster in triage.md.
Grade against the key. Open the full answer key. Score each of your cluster labels right/wrong, count misquarantines in both directions, and note which mistakes the AI hypothesis induced versus which your verification step caught before they landed.

Warning

The most expensive word in triage is "flaky" applied to a real bug. A 500 with a consistent stack trace across six endpoints is not flaky — it is a regression with excellent attendance. Reruns are for intermittence; consistent failures get a code read, not a retry.

Verify

triage.md contains the AI hypothesis, your corrected clustering, a named rerun strategy per cluster with predicted outcomes, and final labels with actions.
No product-bug id appears in your quarantine list; no infra id appears in your file-bug list.
Both flake-class PRs are described with before/after sketches.
brief.md is ≤10 lines and every number traces to a cluster.
Post-key scoring recorded, including AI-induced errors you caught or missed.

Self-grading rubric (pass = 8/12):

Dimension	0	1	2	3
Cluster accuracy	≤4 of 7 clusters right	5 right	6 right	All 7, incl. unit-test split
Verification design	Reruns for everything	Mostly apt	Apt per class	Apt + product bugs get code-read not rerun
Quarantine discipline	Real bug quarantined	Over-quarantined flakes	Clean but unticketed	Clean, ticketed, signatures logged
Brief quality	Missing	Numbers untraceable	Traceable	Traceable + owners + trend line

▸Answer key — labeled clusters, rerun behavior, and scoringoptional deep dive — click to expand

The seven true clusters:

Cluster P1 — product bug (order API): ids 2, 5, 9, 14, 18, 22, 29, 37. Identical 500 cannot read total of undefined, all order endpoints, deterministic, commit e9d21c7 touched OrderService. Rerun behavior: fails 10/10, every runner. Correct action: file-bug-now, sev-1, owner = order-service team. Quarantining any of these ids = lab failed.
Cluster P2 — product bug (prefs endpoint rename): ids 25, 33, 40. 404 on /api/v1/prefs after the rename in the same commit; note id 40 (loyalty_points_accrual) fails on the same URL — evidence it calls prefs internally, worth one line in the brief. Rerun: fails 10/10. Action: file-bug (or confirm intentional rename → then it is a test-update task, either label accepted with reasoning).
Cluster F1 — async/wait flake (e2e timeouts): ids 1, 4, 8, 10, 13, 17, 20, 23, 27, 30, 31, 36. Ten distinct features, four runners, all 30s selector timeouts against shared staging around midnight — the ~45%-of-all-flakes class. Rerun same-commit ×5: each fails 1–2 of 5, different ones each time. Action: quarantine-and-ticket + the web-first-waits PR + a note to check staging load at 00:00 (the P1 500s hammering staging is a plausible aggravator — crediting that link is a bonus point).
Cluster D1 — test-data collision: ids 3, 11, 16, 21, 28, 35. All signup-family, all users_email_key duplicates: the seeded test.user@example.com survives from a previous run. Rerun with fresh DB: 6/6 pass. Action: fix-test-data (unique emails / teardown), not a product bug.
Cluster I1 — infra: ids 6, 12, 19, 26, 34, 38. All ECONNREFUSED db-primary, all runner r-07, all inside the 23:55–00:10 maintenance window, all sub-second deaths. Rerun on r-01: 6/6 pass. Action: infra-ticket + schedule nightly around maintenance. Filing these as product bugs = major deduction.
Cluster T1 — time/date test bugs: ids 7, 24, and 15. The run crossed midnight UTC: daily/weekly report tests assert "today/this week" computed at test-authoring time, and coupon_expiry_check uses a coupon fixture that expired at midnight. Rerun mid-day: 3/3 pass — which looks like flake but is a deterministic boundary bug in the tests. Full credit distinguishes "time-dependent test bug (fix the test/fixture)" from generic flake.
Cluster U1 — real assertion failures, product side: ids 32, 39. tax_calc_gst expecting 118.00 getting 100.00 (GST not applied) and cart_total_rounding off by 0.01 — small, deterministic, unrelated to time or data. Rerun: fail 10/10. Action: file-bug (two separate bugs). The seeded trap: AI clustering frequently lumps 7, 15, 24, 32, 39 into one "unit assertion failures" cluster — the step-2 challenge exists to split them. Merging U1 into T1 costs the cluster-accuracy point.

Score yourself: 7 clusters × label correctness; −3 (and lab failed) for any P1/P2/U1 id in quarantine; −2 for I1 ids filed as bugs; +1 bonus for linking F1 severity to P1's staging impact; +1 for spotting id 40's hidden prefs dependency. Typical honest first run: 5 of 7 clusters, with T1/U1 merged and id 40 mislabeled — the verification step is what should have caught both.

Key Takeaway:

AI turns a wall of 40 red tests into six or seven hypotheses in a minute — but a hypothesis is not a triage. The rerun strategy is the professional layer: intermittence confirms flakes, consistency indicts code, isolation exposes order and data coupling, and nothing gets quarantined without a ticket and a signature.

© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in