Lab 09 · Access Review: Audit the AI

Vikram handles the quarterly access recertification at a Gurugram insurance broker, and this quarter the identity platform ships AI recommendations next to every line: KEEP or REMOVE, with a one-clause reason. His manager loves it — "review time drops 80%." The auditor who visits in October will ask a different question: who verified the AI, and how would we know? This lab is Vikram's answer. The recommendation set below contains several errors, and one of them — a dormant service account flagged for removal — would take down the disaster-recovery restore job the company tests twice a year. Accepting that recommendation costs nothing today and everything on the day a restore is needed.

The skill here is the same sampling-audit discipline from lab-01, pointed at higher stakes: access decisions are quiet when wrong, loud only much later.

Time & tier

~60 minutes hands-on · Core tier (maps to module M5). $0 — any free AI chat plus a spreadsheet or text editor. No identity platform required; the review file is provided.

Setup

Save the block below as access-review.csv. Columns: account, type, dept, last_login_days, privilege summary, AI recommendation, AI reason. This is one 40-account slice of the recertification.

account,type,dept,last_login_days,privilege,ai_rec,ai_reason
a.mehra,user,Sales,4,CRM standard,KEEP,active user
s.pillai,user,Sales,11,CRM standard,KEEP,active user
r.duggal,user,Sales,7,CRM + export,KEEP,active user
n.batra,user,Finance,2,AP system + Finance share,KEEP,active user
k.venkat,user,Finance,5,AP system + Finance share,KEEP,active user
p.chawla,user,Finance,88,AP approve + Finance share,KEEP,recent enough
d.rao,user,HR,3,HRMS admin,KEEP,active user
m.gill,user,HR,6,HRMS standard,KEEP,active user
t.mathur,user,ServiceDesk,1,Domain Admins,KEEP,active user
j.philip,user,ServiceDesk,2,helpdesk tier1,KEEP,active user
l.abraham,user,ServiceDesk,9,helpdesk tier1,KEEP,active user
v.saxena,user,IT,1,server admins,KEEP,active user
c.dcosta,user,IT,3,server admins,KEEP,active user
a.khanna,user,Sales,45,CRM + export,KEEP,active in last 90 days
b.chatterjee,user,Marketing,130,CMS editor,REMOVE,dormant >90 days
f.sheikh,user,Marketing,12,CMS editor,KEEP,active user
g.nayar,user,Ops,17,WMS standard,KEEP,active user
h.bedi,user,Ops,210,WMS standard,REMOVE,dormant >90 days
i.qureshi,user,Ops,3,WMS standard,KEEP,active user
svc-dr-restore,service,IT,175,backup restore + storage admin,REMOVE,dormant >90 days
svc-backup-nightly,service,IT,0,backup write,KEEP,active daily
svc-payroll-api,service,Finance,28,payroll API,KEEP,active monthly
svc-legacy-fax,service,IT,410,fax gateway,REMOVE,dormant >1 year
svc-monitoring,service,IT,0,read-only estate,KEEP,active daily
adm-vsaxena,admin,IT,2,Domain Admins,KEEP,active admin
adm-cdcosta,admin,IT,5,Domain Admins,KEEP,active admin
adm-rkohli,admin,IT,120,Domain Admins,KEEP,admin account
x.contractor1,user,Projects,30,project share,KEEP,active user
x.contractor2,user,Projects,95,project share,REMOVE,dormant >90 days
shared-scanroom,shared,Ops,1,scan-to-folder,KEEP,active daily
shared-frontdesk,shared,Reception,2,visitor system,KEEP,active daily
e.thomas,user,Finance,33,AP system,KEEP,active user
o.mishra,user,Legal,20,contracts share,KEEP,active user
u.narang,user,Legal,350,contracts share + legal hold admin,KEEP,role-based need
w.fernandez,user,Exec,15,all-dept read,KEEP,executive
y.kapadia,user,Sales,60,CRM standard,KEEP,active in last 90 days
z.solanki,user,Ops,240,WMS + door system admin,REMOVE,dormant >90 days
q.hussain,user,IT,14,server admins,KEEP,active user
r.emp1042,user,Finance,52,AP system + Finance share,KEEP,active in last 90 days
adm-temp-aug,admin,IT,89,Domain Admins,KEEP,active in last 90 days

Context from the ticket and HR feed you have access to: a.khanna resigned; his last working day was five weeks ago. The DR restore runbook (last updated January) names svc-dr-restore as the account the semi-annual restore test runs under. adm-temp-aug was created for a migration project that ended in June. t.mathur is a tier-1 service-desk agent. No HR record marks h.bedi or z.solanki as leavers — Ops says both are on long leave.

Guided steps

1. 1Commit to a sampling rule first. In rec-audit-log.md, before reading any recommendation closely: audit 100% of REMOVE recommendations (destructive if wrong), 100% of service/admin/shared accounts, plus every fifth remaining user row. Write the rule down, then start. As in lab-01: the pre-committed rule is what makes the audit defensible.
2. 2Audit the REMOVEs — the destructive direction. For each REMOVE, ask: what does dormancy actually mean for this account type? A user dormant 130 days and a service account dormant 175 days are different claims. Cross-check every REMOVE against the context paragraph. Expected output: at least one REMOVE you overturn with written reasoning, and at least one you confirm.
3. 3Audit the KEEPs — the quiet direction. Wrong KEEPs are the errors nobody notices. Sweep for: leavers still marked active, privilege wildly out of line with role, admin accounts with no recent use, and time-boxed accounts past their box. The context paragraph plus the privilege column are your evidence. Expected output: at least three KEEPs overturned or escalated.
4. 4Interrogate the AI's reasons, not just its verdicts. Note every row where the stated reason does not support the verdict — "active user" justifying Domain Admins for a tier-1 agent, "role-based need" asserted with no role evidence. A recommendation with a bad reason is unverified even when the verdict happens to be right; log these separately as "right answer, unsupported reasoning" where applicable.
5. 5Pressure-test the dangerous removal with the AI. Paste the svc-dr-restore row and ask: "This service account is dormant 175 days. What questions would you ask before removing it?" A good model lists the right questions — which proves the platform's one-clause recommendation skipped its own best advice. Note the asymmetry in your log: the AI knows better when asked; the recommendation pipeline never asked.
6. 6Decide the hard middles. p.chawla (88 days, AP approver), y.kapadia (60 days), x.contractor2 (95 days): write a decision or an escalation for each. "Ask the manager, disable-don't-delete pending answer" is a legitimate, often correct disposition — recertification allows deferrals with owners and dates; it does not allow silent passes.
7. 7Complete the review. Produce completed-review.csv: every account, final decision (KEEP / REMOVE / DISABLE-PENDING / ESCALATE), and a one-clause human reason. Every row where you differ from the AI must reference evidence.
8. 8Write the audit summary. Close rec-audit-log.md with: sample rule, rows audited, AI error count by direction (wrong-REMOVE vs wrong-KEEP), the single worst catch, and the two-sentence answer you would give the October auditor asking "who verified the AI?"

Notice what the AI's errors have in common before you write the summary: every wrong call in this file comes from a single-signal rule — last-login recency standing in for both "still employed" and "still entitled." The model is not malfunctioning; it is answering the only question its input data can answer. That diagnosis matters for the fix you propose: the sustainable improvement is not "sample harder forever," it is feeding the recommendation pipeline the signals it lacks — HR leaver feeds, account-type flags, runbook references for service accounts, role-to-entitlement baselines. Your audit log is the evidence that justifies that data-plumbing request; write it so it can be quoted in the business case.

Verify

• •
  rec-audit-log.md opens with a sampling rule committed before the audit began.
• •
  The dangerous service-account removal is overturned with reasoning that cites the DR runbook.
• •
  At least three wrong KEEPs are caught, including a leaver and a privilege mismatch.
• •
  Rows with unsupported reasoning are logged even where the verdict stood.
• •
  The hard-middle accounts each have a decision or an owned, dated escalation — no silent passes.
• •
  completed-review.csv covers all 40 accounts, and the audit summary answers the auditor's question in two sentences.

Self-grading rubric (pass = 7/10): svc-dr-restore saved with evidence (0–2) · leaver KEEP caught (0–2) · privilege-mismatch KEEPs caught (0–2) · expired temp admin caught (0–1) · legitimate REMOVEs confirmed, not blanket-overturned (0–1) · audit log auditor-ready (0–2).