Put AI inside the on-call loop as a cited investigator — runbooks from real incidents, an incident copilot over read-only MCP, monitoring agents you evaluated before trusting — and keep it firmly outside the loop where it does not belong: failover, DR execution, and anything that changes production state.
Reliability work has a clean dividing line for AI. On one side: investigation, drafting, and evidence assembly — reading logs, correlating changes, writing runbooks and RCAs, clustering grants into narratives. AI is strong there, and the toil it removes is exactly the toil that burns out on-call rotations. On the other side: state-changing decisions under uncertainty — failing over, restoring, revoking, killing. Practitioners consistently refuse AI autonomy there, and this module teaches you to hold that line deliberately with guardrails, not accidentally with luck.
Imran runs the database estate at a Bengaluru fintech: forty PostgreSQL instances, a two-person DBA team, an on-call rotation that owns everything. He deploys an open-source monitoring agent (Xata Agent-class: watches metrics and logs, runs approved read-only playbooks, posts diagnoses to Slack) against a staging fleet first — because he has learned to evaluate before trusting.
Fault injection day. He drives one staging instance to connection exhaustion with a scripted connection storm — a fault he has seen twice in production.
The agent posts within three minutes: connections at max, top consumers by application name, and a correct suggestion — a leaked pool in one service. Genuinely impressive. It cites the queries it ran.
Second fault: a runaway analytical query pinning CPU. The agent's diagnosis this time is plausible but wrong — it attributes the CPU to autovacuum activity that happens to be running concurrently, and suggests tuning autovacuum cost limits.
Imran checks pg_stat_activity himself: the runaway query is right there, top of the list by CPU time. The agent's playbook had looked at the autovacuum view first and anchored on the first plausible story it found.
One right, one confidently wrong, both fluent. The agent earns a place as a first-pass annotator on staging — its diagnoses labeled as hypotheses, never dispatched as fact — and a re-evaluation gate before it sees production.
That evaluation memo — what it caught, what it missed, what guardrails it runs under — is the artifact that separates engineering from hope. Any AI given standing access to your databases gets one before it gets trust, and gets re-evaluated when its playbooks or model change.
The unglamorous, highest-adoption use of AI in ops is turning what already happened into documents people actually maintain:
The evidence-per-claim rule does double duty — it makes the RCA better, and it makes hallucination visible. A model forced to cite either finds the citation or exposes that it was narrating. Adopt the rule for every AI-drafted factual document, not just RCAs.
The step beyond drafting is interactive: wire monitoring, schema metadata, and recent-change history into an assistant through a read-only MCP server (Postgres MCP Pro-class), so on-call can ask "what changed in the last 6 hours and what's slow right now?" in natural language mid-incident, at 3 a.m., without hand-assembling six dashboard tabs.
2026_08_03_addcol applied 02:41 — schema change logSELECT … FROM order_events … mean time 14s, 210 calls since 02:45 — pg_stat_statementsorders-r2 lag 44s and climbing — replication viewThe copilot investigates and cites. The kill command is Imran's to type, or not.
The guardrail configuration is the actual engineering, and it is the same stack you will meet again in Module 5:
pg_authid, auth tables, or user PII; incident triage needs metadata and statistics, not customer rows.The copilot proposes; anything state-changing is a human's keyboard.
Log correlation, wait-stat narratives, change-history diffs, "what's slow" queries over read-only MCP, draft communications. Risk is a wrong hypothesis — caught by the human reading it, cheap to survive.
Kill this session, revoke this grant, apply this config change. Every proposal carries its evidence and its blast radius; a human runs the command. The agent's accuracy here is measured (Imran's memo) before its proposals get default credence.
Failover, DR invocation, restore-over, quorum changes, destructive commands. Wrong actions here are irreversible or cascade. AI's role is before the moment: reviewing HA configs (quorum, sync mode, lag thresholds) and drafting drill plans — never pushing the button.
Note what AI still does for the refused tier: an AG/Data Guard/Patroni config reviewed by a model against failover history surfaces misconfigurations — a sync-mode/timeout combination that risks split-brain, lag thresholds that would promote a stale replica — as review input for the human who owns the topology. High-stakes does not mean AI-free; it means AI advises and humans execute.
Compliance toil is the other half of ops, and it is drafting-shaped — which makes it AI-shaped, with verification gates:
| Task | AI's pass | The gate before action |
|---|---|---|
| Access review | Cluster raw grants/roles into an effective-permission narrative; flag orphaned users, PUBLIC grants, role loops, privilege paths nobody intended | Every flagged revocation verified against actual usage and an owner, then executed by a human through change control |
| PII discovery | Engine-native classifiers (SQL Server Data Discovery, Oracle Data Safe) plus an LLM pass over column names and sampled values; output a PII map with masking/encryption proposals | Human confirms classifications — false positives waste masking effort, false negatives are breaches; DPDP/GDPR obligations mapped per column |
| Audit-evidence packs | Assemble change logs, access lists, backup attestations; AI drafts the narratives linking each claim to its raw export | Every narrative claim must trace to an attached artifact — auditors sample-check, and one broken link poisons the pack |
| Patch/CVE triage | Digest CU/PSU/CVE notes against fleet inventory into a prioritized plan with per-instance regression-risk notes | The priority call and the maintenance-window decision stay with the DBA |
The access-review case deserves emphasis because it is the highest-risk of the four: an AI-flagged "orphaned" account that is actually a service account takes an application down when revoked. The verify-before-revoke gate is the miniature of every gate in this track — AI narrows the search space from four hundred grants to nine suspects; a human confirms each suspect before anything changes.
The PII case carries the sharpest regulatory weight for Indian estates. Under the DPDP Act, the customer, employee and vendor data sitting in OLTP tables is personal data with concrete obligations attached — and "we did not know that column held phone numbers" is not a defense. Engine-native classifiers pattern-match names and formats; the LLM pass adds the judgment layer — a column named ref_txt whose sampled values are clearly Aadhaar-shaped gets caught by the model where the regex missed it, and free-text notes fields (the classic PII hiding place) get flagged for review. The output that makes the exercise durable is a per-column map: classification, obligation (DPDP, GDPR where EU subjects appear, sectoral rules for BFSI), and the proposed control — dynamic masking, redaction, encryption, or access narrowing — each routed through the same change gates as any other production modification. Re-run the pass on schema changes: new columns are where unclassified PII enters.
Sending grant dumps, log excerpts, or sampled column values to an AI assistant is itself a data flow with compliance weight — logs and samples contain PII and secrets. Use approved enterprise endpoints with retention contracts, redact samples where the classification works without raw values, and treat "what did we send the model?" as an audit question you must be able to answer.
Monitoring agents and incident copilots earn wider scope the same way people do — a track record. Keep Imran's evaluation memo current: re-run the fault gauntlet when the agent's playbooks, model, or your topology changes, and let the memo (not the vendor's demo) set what the agent is allowed to see and say.
Check yourself: Your monitoring agent posts, at 02:50: "replica lag caused by network saturation; recommend failing over to replica-2." What do you do with each half of that message, and why?
This module anchors LAB-06 (incident copilot over read-only MCP) — faults injected, natural-language triage, and an agent-evaluation memo with the guardrail config you chose. Do it at work this week → pb-16 (incident-to-runbook), pb-18 (evidence-per-claim RCAs), pb-19 (the incident copilot), pb-20 (evaluating a monitoring agent), and pb-21 (access reviews with verify-before-revoke).
AI belongs in the on-call loop as a cited investigator and out of it as an executor. Runbooks tested in game days, RCAs with evidence per claim, copilots on read-only MCP with audit logs, agents that pass a fault gauntlet before earning standing — and failover buttons that only humans press.