This module installs the habit of shipping a mechanical gate alongside every AI-generated artifact: a mutation score for tests, a CI drift check for docs, a disconfirming check for incident hypotheses. AI supplies the volume; the gate is what makes the volume worth keeping.
After the missing-cent fix, Sana wants billing/proration.py actually covered — it had sat under 30% since she inherited it.
She asks her assistant for tests. It produces forty, and coverage leaps to 86%.
Ever the skeptic, he isn't impressed by the number. "Coverage measures execution, not detection. Run the mutants."
She runs mutmut. Nine survive.
The tool flipped a >= to > at a period boundary — the exact off-by-one that would recompute proration wrong on the last day of a cycle — and the whole forty-test suite passes anyway, because those tests asserted the result was "not None" rather than any specific value.
Coverage theater, precisely: lines executed, nothing checked.
She feeds the survivor back — "this mutant lived; write a test that kills it" — and now a test pins the boundary value.
The line she writes into the team's notes is blunt: a generated suite isn't done when coverage is green, it's done when it kills mutants above the bar. The gate, not the volume, is the deliverable.
There is a version of AI-assisted engineering that looks productive and isn't: a thousand generated tests that assert nothing, documentation that was accurate the day it was generated and never again, and an incident channel where someone pasted a stack trace into a chatbot and restarted the wrong service. This module is about the other version — where AI accelerates the unglamorous ownership work of a senior engineer (tests, flaky-test debt, docs, incidents, postmortems, design reviews) and every artifact it touches comes with a mechanical proof of quality: a mutation score, a drift check in CI, a disconfirming test, a verified timeline.
The pattern that repeats through this whole module: AI generates volume; you supply the gate. Generated tests are cheap, so the differentiator is a quality gate that kills the worthless ones. Generated docs are cheap, so the differentiator is automation that detects when they rot. Generated incident hypotheses are cheap, so the differentiator is the discipline of verifying before touching production. You already learned trust calibration in the Common Core; this module turns it into concrete tooling.
Every AI-generated quality artifact needs a mechanical gate that doesn't depend on anyone's diligence — mutation scores for tests, CI drift checks for docs, disconfirming evidence for incident hypotheses. If the gate is "someone will review it carefully," you don't have a gate.
Around 45% of developers already use AI for testing work (2025 Stack Overflow survey, per our phase-1 research), and test generation is genuinely one of AI's strongest use cases: given a module and its types, a good model produces plausible tests at a rate no human matches. The problem is the word plausible. Left unguided, generated tests tend toward three failure modes:
assert result is not None or assert response.status_code in (200, 400, 500) that are technically assertions and practically decoration.Coverage tools can't distinguish any of these from a good test, because coverage measures execution, not detection. You need a metric that measures detection directly.
Mutation testing answers the question a coverage report can't: if this code were subtly broken, would the suite notice? The tool makes small deliberate changes to your source — flip a > to >=, replace a + with a -, delete a statement — and reruns the tests against each mutant. A mutant your tests fail on is "killed." A mutant that survives means your tests exercised that code without actually checking it.
The workflow for AI-generated tests on a legacy Python module:
# 1. Baseline: check current coverage on the target module
pytest --cov=app/billing --cov-report=term-missing
# 2. Generate tests with your assistant, constrained by prompt:
# "Write pytest tests for app/billing/proration.py. Assert on exact
# computed values, not just types or non-None. Include boundary
# cases at period edges and zero-amount invoices. Do not mock the
# function under test."
# 3. Human pass: delete tautologies and weak oracles before running anything
# 4. Mutation run scoped to the module
mutmut run --paths-to-mutate app/billing/proration.py
mutmut results # list surviving mutants
mutmut show 42 # inspect one survivor's diff
Each surviving mutant is a concrete, actionable finding: here is a specific change to your logic that your entire suite fails to detect. Feed survivors back to the assistant one at a time — "this mutant survived; write a test that kills it" — and you get a tight generate-verify loop where the tool, not your judgment, decides when you're done. On JavaScript/TypeScript, Stryker plays the same role (npx stryker run), and reports a mutation score you can threshold in CI.
Two practical calibrations:
If asked how to validate AI-generated tests, "review them and check coverage went up" is an incomplete answer. The expected answer names mutation testing: generated tests are accepted only when they kill mutants above a threshold, because coverage measures execution while mutation score measures defect detection.
Your first 15 minutes after any AI test-generation run: don't trust the coverage number — run mutmut/Stryker scoped to the changed module and feed one surviving mutant back with "write a test that kills this." Coverage measures execution; the mutant that lives is the assertion you forgot to make, exactly the >=-to-> survivor Sana's 86% hid.
Example-based tests — even good ones — check the points you thought of. Property-based testing (PBT) checks a rule across hundreds of generated inputs, and its historical adoption barrier has been the hard part: figuring out what the invariants are. That elicitation step is exactly what an LLM is good at.
The workflow: paste the function's spec (or docstring, or the ticket) and ask for invariants, not tests — "list properties that must hold for all valid inputs: round-trip laws, ordering guarantees, conservation rules, idempotency, bounds." Then have it express the ones you endorse as Hypothesis properties:
from hypothesis import given, strategies as st
from app.billing.proration import prorate
@given(
amount_cents=st.integers(min_value=0, max_value=10_000_000),
days_used=st.integers(min_value=0, max_value=31),
days_in_period=st.integers(min_value=1, max_value=31),
)
def test_proration_never_exceeds_full_amount(amount_cents, days_used, days_in_period):
result = prorate(amount_cents, days_used, days_in_period)
assert 0 <= result <= amount_cents
@given(amount_cents=st.integers(min_value=0, max_value=10_000_000),
days_in_period=st.integers(min_value=1, max_value=31))
def test_full_period_charges_full_amount(amount_cents, days_in_period):
assert prorate(amount_cents, days_in_period, days_in_period) == amount_cents
When a property fails, Hypothesis shrinks the counterexample to a minimal failing input — which is precisely the artifact you want to paste back into the assistant for a fix hypothesis. On TypeScript, fast-check gives you the same model (fc.assert(fc.property(...))). PBT and mutation testing compose well: properties tend to kill the mutants that example tests miss, because a rule constrains far more of the behavior space than any list of examples.
Your review job in this loop is validating the invariants, not the code. A wrong property generated confidently — "output is always positive" when zero is legal — will fail spuriously and erode trust in the suite. The model proposes; the person who owns the spec disposes.
Ask the model for invariants that are true, and separately for invariants that a careless reader would assume are true but aren't. The second list is where your best regression tests and your documentation clarifications come from.
Two adjacent wins worth folding into the same discipline. Schema-conditioned fixture generation — paste your SQLAlchemy/Prisma models or JSON schema and ask for factory-boy/Faker factories with realistic distributions and the ugly edge cases (unicode names, zero-quantity lines, timezone-crossing timestamps) — replaces the sad user1/test@test.com fixtures most suites accumulate. And spec-driven contract tests — generated from your OpenAPI or proto definitions, including the failure modes clients actually hit (timeouts, 429s, malformed payloads, missing optional fields) — pair well with schemathesis, which fuzzes your API directly from the OpenAPI spec for free. In both cases the spec is the source of truth and the AI is a compiler from spec to test code, which is the trust posture you want.
Flaky tests are a classic senior-engineer time sink: individually cheap to retry, collectively corrosive, and nobody's job. AI changes the economics of diagnosing them because the raw material — CI history, test source, timing data — is exactly the kind of high-volume, pattern-bearing text LLMs digest well.
The discipline is classify before you fix. Flakes fall into a small taxonomy, and each class has a different deterministic fix:
| Class | Signature in CI history | Deterministic fix |
|---|---|---|
| Timing / async | Fails more under load; stack shows sleeps, waits, timeouts | Replace sleeps with explicit condition waits; inject clocks |
| Isolation / shared state | Passes alone, fails in suite; order-dependent | Fresh fixtures per test; reset globals/DB between tests |
| Ordering | Fails only under certain seeds/shuffles | Run with randomized order locally; remove inter-test deps |
| Infra / external | Correlates with specific runners, times of day, third-party outages | Hermetic test doubles; retry only at the infra boundary |
| Resource leakage | Failure rate grows over a run; port/file-handle errors | Teardown audits; leak detection in fixtures |
The workflow: export the flaky test's last 30–50 CI results (pass/fail, duration, runner, failure output), paste them with the test source, and ask for a classification into this taxonomy plus a ranked list of candidate causes with a way to confirm each one cheaply. That last clause matters — "run it 200 times with pytest -p no:randomly vs with ordering randomized" is a confirmation; "it's probably a race condition" is a shrug. Once classified, the fix is usually mechanical and the assistant can draft it; your review focuses on whether the fix removes the nondeterminism or merely hides it (a widened timeout hides; an explicit await removes).
Pair this with a team policy that flakes get quarantined-with-a-ticket rather than retried forever: AI makes the diagnosis cheap enough that "we'll get to it" stops being the honest answer.
Never accept an AI-suggested flaky-test fix that works by adding retries or widening timeouts unless the diagnosis specifically identified an infra-boundary cause. Retries convert a visible flake into an invisible slow test and leave the underlying race in place — often in production code paths, not just the test.
Docs are among the most common AI uses (~61% of developers in our phase-1 research), and also the easiest place to create negative value: a generated doc that drifts out of date is worse than no doc, because readers trust it. The engineering move is to treat documentation as a build artifact with a freshness check, not a writing task.
The generation half is straightforward: point an agent at a module and get docstrings, a README section, or API reference prose; verify it reads true before merging. The half that separates professionals is the drift gate — CI automation that fails when code and docs diverge. Practical gates, cheapest first:
pytest --doctest-modules, or a script that extracts and executes fenced blocks). A renamed function now breaks the build, not the reader.api/ without touching docs/api.md, the PR gets an automated comment asking whether the doc needs an update — annoying exactly as often as it should be.Architecture Decision Records fail in practice because writing them is homework that arrives after the decision energy is spent. The fix: the decision discussion already exists — a Slack thread, a meeting transcript, a PR argument. Paste it with an ADR template (context, decision, alternatives considered, consequences) and have the model draft the record, with one hard instruction: represent the rejected alternatives as their advocates argued them, not as strawmen. Then edit for honesty — you're the one who was in the room. Ten minutes instead of a week of not doing it. Number them, keep them in the repo, and link them from the code they justify; an ADR nobody can find is an ADR that doesn't exist.
Release notes are a summarization pipeline over merged PRs — an almost perfect LLM task, with one wrinkle: audience. Generate two tiers from the same input: an engineer-facing changelog (precise, includes migration notes and breaking-change flags, links PRs) and a customer-facing one (outcome-oriented, no internal jargon, silent on pure refactors). Tools like release-please handle the mechanical accumulation from conventional commits; the LLM layer adds the human-readable tiering. The gate here is factual: every claimed change must trace to a merged PR — instruct the model to include the PR number for each line, and reject any line it can't cite.
Architecture diagrams lie because they're drawn from what someone remembered in a meeting. Generate them from the repo instead: have an agent walk the actual imports, service calls, and queue topology, and emit Mermaid (or C4 via Structurizr DSL) that lives in the repo next to the code. Because the diagram source is text in version control, it participates in review — a PR that adds a new service dependency can be expected to update the diagram, and an AI review pass can flag when it doesn't. Regenerate on a schedule and diff: an unexpected change in the generated diagram is an unexpected change in your architecture.
A generated doc is a liability until it has a drift gate. Ship the check with the doc in the same PR — generation without the gate is how you industrialize stale documentation.
Here is the strongest resistance signal in the 2025 Stack Overflow survey data our research draws on: roughly 76% of developers do not want AI operating in deployment and monitoring. Treat that number as accumulated professional judgment rather than technophobia. Incidents are exactly where LLM failure modes are most expensive: a confident wrong hypothesis at 3 a.m. burns the minutes you don't have, and a wrong action turns an incident into an outage.
So the operating model for AI in incidents is sharply asymmetric: AI reads everything; humans touch everything.
Where AI genuinely helps under time pressure:
f3a91c2, first in the checkout service, only on pods in zone B" — long-context summarization is the mechanical part of triage, and models do it faster than grep-and-scroll.The verification line, stated as rules you can adopt verbatim:
Where your platform supports it (Grafana/Datadog MCP servers, for instance), wiring the assistant directly to observability makes the read side faster; the write side stays human regardless.
The 76% deploy/monitoring caution from the 2025 Stack Overflow survey is the theme of this whole section — the industry consensus is AI-assisted triage with human-gated action, and interviewers probing incident scenarios are listening for exactly that boundary.
Postmortems suffer the same failure as ADRs: high value, written (or not) days later when memory has faded and motivation has moved on. The raw material, again, already exists — the incident channel is a timestamped log of what people saw, tried, and said. The workflow: export the channel, add the alert history and the resolution diff, and have the model produce three things:
Then the human pass, which is where the actual value lives: correct the causal chain (models mistake correlation in chat for causation in systems), enforce blamelessness (strip anything reading as "X pushed the bad config" down to systemic contributors — "config change lacked validation; review didn't catch it because…"), and ruthlessly cut the follow-up list to what will really be staffed. A postmortem drafted within an hour of resolution, while everyone's memory is hot, gets reviewed and actioned; one drafted next Thursday gets filed.
Every resolved incident should mint or update a runbook — and with the postmortem already drafted, the marginal cost of "extract the diagnostic steps and mitigation into a runbook entry with exact commands" is minutes. The harder problem is rot: runbooks reference dashboards that moved, services that renamed, commands whose flags changed. Apply the drift-gate pattern from the docs section: on a schedule, an agent walks each runbook and checks its claims against current reality — do the named services exist, do the dashboard links resolve, do the commands parse against current CLI versions, do the config paths still exist — and opens a PR flagging every stale reference, with proposed fixes for the mechanical ones. That's the honest version of "self-healing": the runbook doesn't fix itself unsupervised; it detects its own decay and files the repair for human review. A runbook with a green freshness check from last Tuesday is one you can trust at 3 a.m.; one nobody has read since it was written is a trap.
Add a "verified against production on DATE by METHOD" line to every runbook, maintained by the drift-check automation. The absence of that line tells an on-call engineer more than the runbook's content does.
The last ownership skill in this module is using AI to attack your own designs before your reviewers — or production — does.
Draft the design yourself. This isn't ceremony: the evidence pattern across our research is that AI adds most value critiquing structured thinking, and least value substituting for it — and a design you didn't author is a design you can't defend in review. Then run four adversarial passes, each with an explicit persona and a demand for concrete failure scenarios rather than themes:
Expect a mix: some findings you've handled (say so in the doc — reviewers will raise them too), some noise, and reliably one or two genuine gaps — the "edge cases I hadn't thought of" pattern practitioners consistently report. Fold the real ones into the design with a change log; the v1 → attack → v2 trail itself is persuasive material in design review, because it shows the design has already survived contact with an adversary.
Formal threat modeling rarely happens for a single new endpoint; a 20-minute AI-drafted STRIDE pass can. Give the model the endpoint spec, auth model, and data flow, and ask for a STRIDE table — Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege — with concrete threats per category, each rated for exploitability and mapped to a proposed mitigation. Your job is validation and triage: kill the generic entries ("an attacker could DoS the endpoint" — yes, and?), keep the specific ones ("the export parameter accepts a URL and the server fetches it — SSRF against your metadata endpoint"), and turn the accepted mitigations into actual tickets in the sprint, because a threat model whose mitigations never reach the backlog is a document, not a control. This composes with M5's security-review pass: the threat model tells your review bots what to watch for in every subsequent PR touching that surface.
Scenario questions about AI and design quality look for the ordering: human drafts, AI attacks in structured passes, human triages findings into a revised design with a change log. "Ask the AI to design it and review the output" gets partial credit at best.
Check yourself: Coverage on a module jumped from 20% to 85% after an AI test-generation run. Why is that number not evidence the tests are any good, and what metric is?
This module is backed by three hands-on labs in the track: Coverage lift with quality gate (raise a legacy module past 80% coverage and prove assertion quality with a mutation-score threshold), Design red-team (draft a rate-limiter design, run the four-lens attack, ship a revised v2 with change log), and Incident-to-postmortem (triage a simulated incident bundle — alerts, logs, deploy history, chat export — to a verified root cause, then produce the blameless postmortem and three runbook updates, graded on evidence discipline). The accompanying playbooks give you copy-adaptable prompt templates for the mutation loop, the flake classifier, the ADR extractor, the incident triage protocol, and the STRIDE pass — each with the restricted-environment variant for client-codebase work. Do this at work this week → pb-27 Coverage Lift on a Legacy Module — With a Mutation-Testing Quality Gate (the gate Vikram made Sana run), pb-30 Flaky-Test Forensics: Classify the Flake, Fix the Cause, Prove It (classify before you fix, from CI history), and pb-38 Incident Triage Copilot: Ranked Root-Cause Hypotheses Under Time Pressure (AI reads everything, humans touch everything).