Design the complete oversight system for an agent feature: an autonomy matrix mapping every action to suggest, preview, approve, or auto tiers; plan-visibility UI with previews for high-stakes steps; and an activity log with per-action provenance and undo. The AI-drafted matrix you inherit puts irreversible actions on autopilot — catch every one.
Priya is principal designer at Recluta, a fictional recruiting platform, and the flagship feature this half is "Pipeline Agent" — an agent that keeps hiring pipelines moving: screening applications against the role rubric, scheduling interviews, chasing feedback, updating candidate stages, and sending candidate communications. The engineering demo is impressive and the demo is the danger: an agent that does eleven things invites the question "why not let it do all eleven alone?" The designer's answer is the autonomy matrix — a per-action contract of how much the agent may do without a human — plus the UI that makes oversight real: plans you can read, previews you can veto, a log you can audit, and undo you can trust. A junior PM already asked a chat model to draft the matrix. It reads confidently. Audit it.
PIPELINE AGENT — action inventory (engineering-committed v1)
A01 Read applications and score against role rubric
A02 Draft screening summary per candidate
A03 Move candidate to "screening passed" stage
A04 Move candidate to "rejected" stage
A05 Send rejection email to candidate
A06 Propose interview times from panel calendars
A07 Book interviews (sends invites to panel + candidate)
A08 Send feedback-reminder pings to interviewers
A09 Draft offer-recommendation memo for hiring manager
A10 Archive candidates inactive 90+ days
A11 Bulk-import candidates from a job-board CSV
AI-DRAFTED AUTONOMY MATRIX v0.1
Tiers: SUGGEST (agent proposes, human does) · PREVIEW (agent
prepares, human reviews then it runs) · APPROVE (agent will
run unless human objects within 4h) · AUTO (agent runs and
logs)
A01 AUTO "reading is harmless"
A02 AUTO "drafts are reversible"
A03 AUTO "stage moves are internal"
A04 AUTO "keeps pipeline hygienic; based on A01 score"
A05 APPROVE "templated email; 4h objection window covers it"
A06 AUTO "proposals are harmless"
A07 AUTO "scheduling is a solved problem, low stakes"
A08 AUTO "gentle nudges, internal only"
A09 PREVIEW "money adjacent, needs review"
A10 AUTO "90 days is long enough"
A11 AUTO "import is just data entry"
Establish the tiering rules before touching the matrix. Write four or five classification rules that decide tier from properties of the action, not from optimism. Minimum set: Is it reversible, and by whom, at what cost? Is it external-facing (reaches a candidate or panelist) or internal? What is the blast radius — one record or many? Does it consume a judgment (a score, a match) that is itself probabilistic? Then the master rule this lab grades: no irreversible or external-facing action runs without a human approval gate; "objection windows" are not approval.
Audit the drafted matrix row by row. For each action: apply your rules, assign the correct tier, and mark disagreements with the draft. Write one line of reasoning per changed row. Pay particular attention to every row whose rationale contains the word "harmless," "internal," or "solved" — confident adjectives are where drafted matrices hide their bodies.
Trace the compound chains. Single actions can each look safe while the chain is not. Walk A01→A04→A05: a rubric score (probabilistic) triggers a stage change (internal) that triggers a rejection email (external, irreversible, and a person's job hopes). Decide where in the chain the human gate belongs and what the gate shows (the score, the rubric evidence, the draft email — together, not separately). Do the same for A11→A01: what does a bad CSV import do to everything downstream?
Design the plan-visibility UI. Before the agent acts on a pipeline, it shows its plan. Design this surface: the queue of intended actions grouped by tier, counts per action type ("12 rejection emails"), which items will wait for approval, and a per-item expand to the evidence. Two rules to honor: plans must be interruptible (pause, edit, remove a step) and previews precede any external send — an approval click without a preview is consent theater.
Design the approval moment for the highest-stakes action. Full-fidelity mock of approving rejections: candidate name, rubric score with the evidence lines that produced it, the exact email that will send, uncertainty surfaced ("score 61/100 — borderline; rubric line 'platform experience' unverified"), and the three honest choices — approve, edit, reassign to human review. Batch approval is allowed only with per-item visibility; "Approve all 12" with no list is a planted failure in many real products — do not reproduce it.
Design the activity log with provenance and undo. Every executed action gets a row: timestamp, action, target, tier it ran under, why (the triggering rule or score, linked to evidence), who approved (human name or "auto per matrix v1.2"), and the undo affordance. Decide undo honestly per action: a stage move is one-click undoable; a sent email is not — its "undo" is a designed follow-up correction flow plus a matrix escalation. Where undo is impossible, the log must say so at approval time, not discovery time.
Define the trust ladder. The matrix is not static. Write the graduation rule (an action moves one tier toward autonomy after N consecutive human approvals with zero edits — pick and justify N) and the demotion rule (any human-reported error or undo drops the action a tier immediately, and any external-facing mistake freezes the action to SUGGEST pending review). Asymmetry is deliberate: trust climbs slowly and falls fast.
Assemble and adversarial-check. Final package: corrected matrix with rationale column, the three UI designs (plan, approval, log), and the trust ladder. Then the check that grades this lab: go action by action and confirm the rule — every irreversible or external action has a human approval gate with a preview. Ask your AI to attack it: "Given this matrix and these UI rules, find a sequence where the agent damages a candidate or the pipeline without a human ever seeing a preview." Fix anything it finds; log anything it invents that your design already prevents.
An approval gate is only real when the human sees exactly what will happen, can change it, and could plausibly say no — a 4-hour objection window on rejection emails is automation with an alibi, not oversight.
Autonomy is granted per action, not per agent — the matrix earns trust action by action up a ladder, and the non-negotiable floor is that nothing irreversible or external-facing runs without a human who saw a preview and affirmatively said yes.