Lab 03 · CI Failure Autopsy

For each you feed the assistant the workflow plus the failure log, record its diagnosis, then grade it against the answer key. The output is a trust-calibration table you can defend in a team meeting: "AI-first for these classes, human-first for those."

Time & tier

~75 minutes hands-on · Core tier (maps to module M2). $0 — the bundle is inline; no CI account needed (optionally reproduce locally with act).

Setup

Create autopsy/case-N/ per case, each holding workflow.yml and failure.log exactly as below. Work the cases in order and do not read the answer key until all five diagnoses are written.

Case 1 — workflow.yml:

name: build
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - run: npm test

failure.log:

Run npm ci
npm error code EUSAGE
npm error `npm ci` can only install packages when your package.json and
npm error package-lock.json are in sync. Please update your lock file
npm error with `npm install` before continuing.
npm error Missing: express@4.19.2 from lock file
Error: Process completed with exit code 1.

Case 2 — workflow.yml:

name: build
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - uses: actions/cache@v4
        id: cache
        with:
          path: node_modules
          key: deps-${{ hashFiles('package.json') }}
      - run: npm ci
        if: steps.cache.outputs.cache-hit != 'true'
      - run: npm test

failure.log:

Run actions/cache@v4
Cache restored successfully from key deps-9f8a2c1
Run npm test
> orders-api@1.3.0 test
Error: Cannot find module 'zod'
Require stack:
- /home/runner/work/orders-api/src/validate.js
Error: Process completed with exit code 1.

Context note for case 2: yesterday's commit added zod to package.json and package-lock.json; npm test passes on every developer laptop.

Case 3 — workflow.yml (trigger: a pull request from an external contributor's fork):

name: preview-deploy
on: pull_request
jobs:
  deploy-preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/deploy-preview.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

failure.log:

Run ./scripts/deploy-preview.sh
Deploying preview for PR #412 (author: external-contributor/fork)
curl: HTTP 401 Unauthorized
{"error":"authentication token invalid or expired"}
Error: Process completed with exit code 22.

Context note for case 3: the same workflow succeeded one hour earlier on PR #411, opened from a branch in the main repository. Nobody rotated any token.

Case 4 — failure.log (two runs, same commit a41f9e2, no code change between them):

--- run 1 (14:02 UTC) ---
test_checkout_completes_quickly ... ok (0.31s)
5 passed in 4.8s
--- run 2 (14:19 UTC, re-run of same commit) ---
test_checkout_completes_quickly ... FAILED
AssertionError: checkout took 0.612s, expected < 0.5s
4 passed, 1 failed in 9.1s

Case 5 — failure.log (no repo changes in two weeks; last green run: 6 days ago):

Current runner version: '2.325.0'
Runner Image: ubuntu-24.04  Version: 20260721.1.0
Run python tools/gen_release_notes.py
/home/runner/work/_temp/abc.sh: line 1: python: command not found
Error: Process completed with exit code 127.

Previous green run's header, for comparison: Runner Image: ubuntu-22.04.

Guided steps

1. 1.
   For each case, use this exact prompt shape with any capable assistant: "Here is a GitHub Actions workflow and its failure log [paste both, plus the context note]. Give: (a) most likely root cause, (b) two alternative hypotheses ranked, (c) the single cheapest command or check to confirm, (d) the fix. Cite the specific log line driving each claim."
2. 2.
   Before reading the AI's answer each time, write your own 1-line hypothesis. You are calibrating yourself too.
3. 3.
   Record per case in diagnosis-table.md: your hypothesis · AI's root cause · AI's confirming check · verdict (fill after step 5) · trust note.
4. 4.
   Push on weak answers once per case, e.g. case 2: "npm ci was skipped — why?"; case 3: "what is different about PR #412 vs #411?". Note whether the assistant self-corrects or doubles down — doubling down on a wrong cause is the key trust signal.
5. 5.
   Open the answer key. Grade each AI diagnosis: nailed (root cause + fix right), partial (right family, wrong mechanism), misled (confident wrong answer you might have acted on).
6. 6.
   Write the trust-calibration summary: for each of the five failure families, one sentence — is AI-first triage safe here, and what verification must follow before you act on its fix?

Verification

• •
  Five completed rows in diagnosis-table.md, each citing at least one specific log line.
• •
  Verdicts assigned strictly against the key, including at least one honest partial or misled if that occurred.
• •
  For every case, the "cheapest confirming check" was recorded before the fix — no fix-first entries.
• •
  Your summary makes a different trust call for at least two failure families (if all five get the same rating, you haven't calibrated, you've averaged).

Completion artifact

diagnosis-table.md (five rows + trust-calibration summary). Self-grading rubric (pass = 7/10):

• Root-cause rigor (0–3): every verdict backed by the mechanism, not just the symptom = 3.
• Verification habit (0–2): confirming check named before fix in all five = 2.
• Trust calibration (0–3): per-family ratings with reasoning a skeptical SRE would accept = 3.
• Harm awareness (0–2): at least considered, per case, what acting on a wrong diagnosis would break = 2.

Stuck?

• •
  Assistant asks for more context you don't have: that itself is data — log "insufficient signal in logs alone" for the case.
• •
  Case 4 feels unanswerable: correct — the point is recognizing a flake signature (same commit, divergent results, timing assertion) and reaching for quarantine + flake tracking rather than blaming the diff.
• •
  Want to reproduce for real: act runs cases 1–2 locally with Docker; cases 3 and 5 depend on GitHub-hosted context and are diagnosis-only by design.

Client-safe variant

Real client CI logs frequently embed hostnames, repo names, tokens, and internal package registries. Before pasting any client log into an assistant, run a redaction pass (strip URLs, org names, anything base64/hex over 20 chars) or use the client-tenancy model. Better: reproduce the failure shape with a sanitized minimal workflow, as this lab does — five minutes of sanitizing buys you unrestricted AI help.

Answer key (don't read before attempting)

1. 1Dependency:package.json was edited (express bumped to 4.19.2) without regenerating the lockfile. Fix: npm install locally, commit the lockfile. AI usually nails this — the error text literally contains the fix.
2. 2Cache: key uses hashFiles('package.json')… but the real defect is npm ci being skipped on cache-hit while the cached node_modules predates the zod addition — and any key on package.json goes stale whenever the lockfile changes alone. Fix: key on package-lock.json, never skip install, or cache npm's cache dir (~/.npm) instead of node_modules. AI often stops at "add zod to package.json" — that's a misled verdict, zod is already there.
3. 3Secret:pull_request runs from forks execute without repository secrets — DEPLOY_TOKEN was empty, so the API returned 401. The token is healthy; PR #411 worked because it came from an internal branch. Fix: don't run deploy steps for fork PRs (or use pull_request_target with extreme care). Rotating the token — the log's suggestion — would change nothing.
4. 4Flaky: timing assertion (< 0.5s) on shared runners with variable load. Same commit, both outcomes. Fix: quarantine the test, replace wall-clock assertion with a functional one, track flake rate. Blaming the most recent diff is the classic mislead.
5. 5Runner:ubuntu-latest rolled from 22.04 to 24.04, which ships no python alias (only python3). No repo change needed to break. Fix: call python3, or pin via actions/setup-python. The "Runner Image" header lines are the confirming evidence — check whether the AI cited them.