Audit AI Access-Review Recommendations

Tier: Advanced · Workflow area: Identity, endpoint & patch · ~40 minutes

This playbook operationalizes AI-assisted recertification — Entra Access Review recommendations or the Access Review Agent — with the sampling-audit discipline that catches wrong-removal and wrong-retain recommendations before they become outages or audit findings. (UC-32)

When to use

• Quarterly or semi-annual recertification campaigns where reviewers face hundreds of decisions and historically approve-all just to clear the queue.
• When you enable Entra's recommendations or the Access Review Agentthe audit habit starts with the first campaign, calibrated on your data, not the vendor's benchmarks.
• Groups mixing humans and non-humansservice accounts, shared mailboxes and break-glass identities inside reviewed groups are exactly where inactivity-based recommendations misfire.

When NOT to use

• As a reason to skip cleanupif a group's membership is 60% unknown, run the hygiene project first; AI recommendations on a garbage group produce confident garbage.
• Fully auto-applying decisions on privileged scopesadmin roles, PIM-eligible groups and anything with tenant-wide reach get human decisions per item, with AI as advisor only.
• When no one owns the exception listan unowned "do not remove" list rots into a bypass mechanism. No owner, no automation.

The recipe

1. 1.

   Build the exception list before the first campaign. Inventory identities that will look dormant but must survive: service accounts with seasonal jobs (quarter-end, year-end), break-glass accounts (never used, by design), leavers-on-legal-hold, integration accounts authenticating with certificates that some reports miss. For each: identity, why it looks dormant, real owner, review cadence. This list is the single highest-value artifact in this playbook.

2. 2.

   Configure the review with recommendations on, auto-apply off. In Entra: Identity Governance > Access Reviews, enable decision helpers (last sign-in, peer outlier analysis); if you are piloting the Access Review Agent, let it prepare recommendations and draft decisions. Leave "auto-apply results" off for at least the first two cycles — results apply only after your audit gate.

3. 3.

   Pull the recommendation set and stratify a sample. Do not sample uniformly; stratify where the failure modes live:

   Copy

   Sample plan per campaign batch:
   - 100% of recommendations touching anything on the exception list
   - 100% of "remove" recommendations for service accounts / non-humans
   - 10 random "remove" recommendations for standard users
   - 10 random "approve/retain" recommendations (wrong-retain is the
     silent failure everyone forgets to check)
   - 100% of items where the reviewer overrode the recommendation
   

4. 4.

   Grade the sample with a structured second opinion. For each sampled item, export the evidence (last interactive sign-in, last non-interactive sign-in, sign-in logs excerpt, group purpose) and ask:

   Copy

   Recommendation: REMOVE user svc-billing-batch from SG-ERP-Prod-Access.
   Evidence: [paste last sign-in data, auth methods, group purpose].
   Answer three questions:
   1. Does the evidence actually support removal? Distinguish "no
      interactive sign-ins" from "no authentications of any kind" —
      check for non-interactive, service principal, and certificate auth.
   2. What breaks if this removal is wrong, and when would we find out?
   3. What one additional check would settle it?
   Be adversarial to the recommendation, not agreeable.
   

   The interactive-vs-non-interactive distinction is the classic catch: several review data sources weight interactive sign-ins, and a batch service account may show none while authenticating nightly.

5. 5.

   Log every audit verdict. The sampling-audit log has six columns: item, recommendation, verdict (correct / wrong-remove / wrong-retain / insufficient-evidence), evidence checked, action, pattern note. Keep it per-campaign in one sheet; it is both your calibration data and your audit evidence.

6. 6.

   Gate the apply step on the audit result. Set a threshold in advance — e.g., wrong-recommendation rate under 5% in the sample and zero wrong-removes on non-humans. Pass: apply the batch. Fail: apply only the human-verified subset, and take the failure pattern to the configuration (data sources, inactivity window, group scoping) before the next batch.

7. 7.

   Trend it across campaigns. After three cycles you have a real error rate. Falling and low: extend sampling intervals and consider auto-apply for the lowest-risk scopes (standard users, non-privileged groups) — never for the exception-list scopes. Flat and high: the recommendations are theater for your environment; say so with the log as proof.

Make it yours

• Entra Access Review Agent shopthe agent drafts recommendations and can act on your behalf once approved; keep its scope to non-privileged groups initially and review its activity log weekly alongside the sampling audit.
• SailPoint / Saviynt shopsame pattern, richer certification data; stratify samples by the platform's own risk score, and audit hardest where the score and your intuition disagree.
• No IGA toolingexport group membership plus last-logon data, have AI produce the recommendations and run the same audit against them; the discipline transfers even when the recommender is a prompt.

Quality check

• Non-humans never get bulk-removed on inactivity evidence alone. Every service-account removal requires positive confirmation from the named owner.
• Both failure directions get sampled. Wrong-retain quietly preserves privilege creep; audit approves, not just removals.
• The exception list has a review date on every row. An exception without an expiry is a permanent hole; recertify the list itself annually.
• Overrides feed calibration. When reviewers repeatedly override the same recommendation class, either the config or the reviewers are wrong — the log tells you which, and either way you act on it.