Cluster Nightly CI Failures into a Morning Triage Brief

Tier: Core · Workflow area: Triage, flakes & defect intelligence

This playbook builds the morning ritual: export overnight failures, have an assistant cluster them by shared signature into the three buckets, verify the risky clusters by hand, and publish a triage brief the team can act on before standup.

Prerequisites

• •
  A machine-readable failure export: JUnit XML, Allure results, or your CI API's per-test failure messages and stack traces.
• •
  An assistant able to take the full export (chunk if needed) — Claude/ChatGPT class.
• •
  A place the brief lands every morning: Slack channel, dashboard, or wiki page.
• •
  Named rotation for the triage duty (the brief needs a human signature).

The workflow

1. 1.

   Export, don't screenshot. Pull the structured failure list with messages and traces:

   Copy

   # example: collect junit xml from last night's run artifacts
   gh run download --name junit-results -D /tmp/nightly && \
   python scripts/failures_to_json.py /tmp/nightly/*.xml > /tmp/failures.json
   

   Include per-test: name, file, failure message, first stack frames, duration, agent/shard, and retry outcome if retries ran.

2. 2.

   Prompt for clustering into the three buckets. Signature first, bucket second, evidence always:

   Copy

   Attached: 40 test failures from last night (JSON). Cluster them:
   1) Group tests whose failure SIGNATURE is shared — same exception type,
      same message shape, same failing endpoint/selector, same shard.
   2) Label each cluster PRODUCT-BUG / TEST-BUG / INFRA with reasoning.
      Rules: many unrelated features failing identically at the same time
      suggests INFRA; failures only in tests touched by yesterday's commits
      suggest TEST-BUG; consistent failures matching a feature area with a
      recent app deploy suggest PRODUCT-BUG.
   3) For each cluster: representative failure quote, member count, member list,
      confidence, and the ONE verification step a human should do first.
   Output as a table ordered by (product-bug first, then size).
   

3. 3.

   Read the draft skeptically — you are the second clusterer. Expect a good first cut and specific errors: two distinct bugs merged because both say TimeoutError, or an infra cluster hiding one real product failure with a coincidentally similar message. Sample draft row:

   Copy

   CLUSTER C2 · 11 tests · INFRA (confidence: high)
   Signature: "ECONNREFUSED payment-sandbox:8443" across checkout, refunds, invoices
   Verify first: curl the sandbox healthcheck; check sandbox deploy log 01:00-03:00
   

4. 4.

   Verify the high-stakes clusters by hand. Product-bug clusters gate releases and page developers — confirm one representative failure per cluster yourself (rerun it, read its trace) before the brief asserts it. Infra clusters get the model's proposed one-step check (curl the dependency, check the runner pool). Test-bug clusters get cross-referenced against yesterday's suite commits.

5. 5.

   Publish the brief — one screen, owners attached.

   Copy

   NIGHTLY TRIAGE — Tue 05 Aug · run #2031 · 40 failures -> 4 clusters
   ● PRODUCT (1): C1 order-total rounding, 6 tests, repro'd by hand,
     matches deploy 22:14 of pricing-svc -> BUG-3312 filed, owner @pricing
   ● INFRA (2): C2 payment-sandbox down 11 tests (sandbox restarted, rerun green);
     C3 runner-pool OOM 19 tests -> ticket INFRA-88, reruns queued
   ● TEST-BUG (1): C4 stale fixture after schema migration, 4 tests -> PR #501
   Signed: <triage duty engineer>
   

   Every failure is in exactly one cluster; every cluster has a next action and a name.

6. 6.

   Feed the loop. File the product bugs (defect-report playbook), send test-bug clusters to fix PRs, recurring infra clusters to platform tickets, and any "flaky" leftovers into the failure-signature diagnosis protocol. Track cluster recurrence week over week — a signature that returns nightly is a systemic issue wearing a daily disguise.

Variations

• ReportPortal / Allure TestOps: let the platform's ML grouping form clusters, and use the assistant only for bucket-labeling, verification proposals, and writing the brief — less token volume, same ritual.
• Two-stage for big suites: first pass clusters on failure messages alone (cheap), second pass sends only ambiguous clusters' full traces for refinement.
• Retry-aware clustering: if retries run overnight, feed retry outcomes in — "failed then passed" concentrates the flake-suspect cluster automatically.
• Slack-bot delivery: a scheduled job runs export → clustering prompt → posts the draft to the triage channel; the duty engineer verifies and edits before pinning. The bot never posts as final.

Quality check

• Coverage: cluster member counts sum to the failure count; no orphan failures silently dropped by the model.
• Evidence per cluster: each has a quoted representative failure; spot-check that quoted lines exist in the real export (models will paraphrase).
• Verification trail: product-bug clusters show a human repro note; infra clusters show the check that was run.
• Action completeness: every cluster ends in a ticket, PR, rerun, or explicit "watch" — a brief without actions is a weather report.
• Recurrence tracked: cluster signatures compared against the previous week; repeats flagged as systemic.

Pitfalls

• Message-similarity myopia. Two different bugs both say expected 200 got 500; clustering on message alone merges them. Endpoint, feature area, and timing belong in the signature.
• The hidden product bug inside the infra cluster. When a dependency dies at 2 a.m., a real regression deployed at 1 a.m. hides in the same red. After infra recovers, rerun the cluster — anything still red gets promoted, not forgotten.
• Unverified briefs going out. One confidently wrong product-bug call and developers stop trusting the ritual; the human signature line exists to keep the incentive honest.
• Bucket inflation toward test-bug. "Probably the test" is the lazy label that lets product bugs age. Test-bug requires positive evidence (suite commit, fixture drift), not absence of other explanations.
• Triage without a duty owner. A brief everyone reads and no one owns decays into wallpaper in two weeks.

Restricted-environment variant

Failure exports carry internal URLs, service names, and sometimes customer data embedded in assertion messages — scrub with the same masking pipeline used for test data before sending to external models. On client engagements, prefer client-approved tooling or a local model for the clustering pass; the taxonomy rules in the prompt keep even mid-size local models useful. The brief itself often goes in client-visible channels: keep it evidence-based and tool-agnostic ("clustered and verified" — the client cares about the verification, not the clustering engine).

Success criteria

• •
  Morning triage time drops from an hour of log-reading to ≤15 minutes of verification and publishing.
• •
  Briefs ship before standup with 100% failure coverage and a named signer, five days a week.
• •
  Zero misrouted product bugs (developer paged for infra noise, or regression sat in the test-bug pile) over a month.
• •
  Recurring clusters trend down as systemic fixes land — the brief should be shrinking.