An agent doesn't just answer — it acts, and every act is a trust decision made on the user's behalf. This module designs the oversight that makes autonomy grantable: autonomy tiers, plan visibility, interruption and undo, handoffs in both directions, calibrated confidence display, and a distinct state for every way the agent can fail.
Arjun is the lead designer on Relay, a field-service product whose new agent can reschedule visits when a technician runs late — recalculating the route, updating the calendar, and emailing affected customers. In the demo it's magic. The team ships it with a single setting: Autopilot, on/off.
On a Tuesday in monsoon season, a technician's phone loses signal for an hour, the agent misreads the stale location data as a five-hour delay, and it does exactly what it was built to do: reschedules nineteen appointments and emails nineteen customers apologies for a delay that isn't happening. There is no log the dispatcher can read to see what happened, no undo, and no way to interrupt the run once it starts — support reconstructs the mess from customer replies. By Thursday, every dispatcher has switched Autopilot off. Usage of the agent: zero. The capability was real; the oversight was missing, and without oversight, users rationally choose the off switch.
The lesson generalizes to every agent feature you'll design: autonomy is not a feature you grant yourself — it's a feature users grant you, and they grant it only when they can see what the agent intends, watch what it does, stop it, and reverse it. This module is that oversight system, pattern by pattern.
Relay's on/off design forced users to choose between total delegation and none. The fix is an autonomy matrix: enumerate every action the agent can take, and assign each to a tier — per action, not per feature.
Agent drafts the reschedule plan and the emails; nothing executes. The dispatcher reviews everything. Zero risk, full labor — this is where new agent features and new users start.
Agent stages the full plan — route, calendar changes, drafted emails — and executes only on explicit approval. One click covers the batch; high-stakes items (customer-facing email) show sample responses before send.
Reversible, well-proven actions (calendar shuffles) execute autonomously — visible in the activity log, undoable, with notification digests. Irreversible actions (sending email) stay gated by rule, regardless of track record.
Two principles govern the matrix. Progressive delegation: autonomy is earned through approval history — a user who has approved fifty reschedule plans can be offered auto-mode for that action class, which converts trust from a leap into an accumulation. Asymmetry by reversibility: what can be undone may graduate; what cannot (sent emails, payments, deletions with legal weight) keeps its approval gate forever. Had Relay shipped this matrix, the monsoon incident would have been a preview screen a dispatcher glanced at and rejected in four seconds.
Before an agent runs, the user should be able to answer: what is it about to do? The Governor patterns from Shape of AI supply the anatomy:
Mid-run control is what makes users willing to start a run at all. The control set:
Specify what survives an interrupt: completed actions stand and appear in the log; staged-but-unexecuted work is preserved as a resumable draft, not lost.
The reverse channel needs equal design: how the agent asks for help. An agent that never asks barrels through ambiguity (Relay's failure); one that asks constantly is a chore in disguise. The heuristic: interrupt for ambiguity that changes outcomes materially, batch the rest into one consolidated question set, and always attach the context needed to answer in one glance — never "should I proceed?" with no plan attached.
If plans are the agent's promises, the log is its accountability — the single pattern that would have most changed Relay's Tuesday.
The same incident, redesigned: stale input visible, emails never sent, one-click recovery.
Design rules for the log: it is persistent and readable — plain language, not telemetry; every action carries provenance (what input and reasoning produced it — the "footprints" pattern, which is how the dispatcher spots "stale ping" at a glance); undo is per-action and per-run for everything reversible; and irreversible actions get approval gates by rule — a decision made in the design system once, not renegotiated per feature. Practitioner findings are unambiguous: a visible audit log with prominent undo dramatically lowers the perceived risk of granting autonomy. It's also what makes progressive delegation legible — the log is the track record.
Agent-to-human escalation triggers on three signals: confidence below the bar for the action class, policy triggers (regulated content, spend thresholds, anything on the irreversible list without a gate decision), and user-frustration signals — repeated corrections, abandoned runs, sentiment shifts. The escalation message is itself a designed artifact with a required anatomy: what was attempted, what's known so far, what's needed from the human, and what happens after. "Something went wrong, please take over" fails that spec; "I couldn't confirm the customer's address — two candidates below; pick one and I'll finish the reschedule" passes it.
Human-to-agent delegation is the mirror image: context transfer at handoff — what the user has done so far travels with the task, so the agent never makes the user repeat themselves to their own software.
For background and multi-agent work, the design problem is attention: status must be visible without demanding polling (ambient state on the task, not a dashboard users must remember to check); completions arrive as digests by default, interrupts by exception — reserved for blocked runs and finished work the user is actively waiting on; the results-arrival moment gets designed like a landing page (what changed, what needs review, what was assumed); and when multiple agents contribute, attribution stays legible, because "which agent did this?" is the debugging question every power user eventually asks.
Every agent output carries uncertainty; the design decision is how — and whether — to show it. Four display strategies, each fit for different stakes: numeric scores (only when users can act on the number; false precision harms), hedged language ("this looks like…" — matches how humans signal doubt, but must map to real model confidence or it's theater), citations (convert trust from belief to verification — the strongest option when sources exist), and abstention (below the bar, say "I can't determine this" — the most honest display of all). The goal is calibrated trust: users relying on the system exactly as much as its accuracy warrants. Designing for maximum trust is a dark pattern; Module 8 measures the difference.
Beneath confidence display sits the module's most practical tool — the AI error taxonomy. "Error state" is not one state; each failure mode needs distinct handling:
| Failure mode | What the user sees without design | The designed state |
|---|---|---|
| Wrong but plausible | A confident answer that's false — no signal at all | Verification affordances: citations, provenance, "check this" prompts — never an error banner (there's no trigger for one) |
| Stale | Correct-sounding answer from outdated data | Data-freshness timestamp; refresh affordance |
| Partial | Half an answer presented as whole | Explicit scope statement: "3 of 5 sources reachable" |
| Refused | A wall, often preachy | Boundary explained, nearest safe alternative, human route (M6's refusal spec) |
| Timeout | Spinner, then nothing | Preserved partial work, resume affordance, honest status |
| Hallucinated citation | A source link that 404s or misattributes | Dead-link detection, downgraded confidence display, report affordance |
Wrong-but-plausible is the row that matters most — it's the failure with no symptom, so the only defense is affordances that make verification cheap. Lab 13 hands you a feature and its real eval-failure list and has you design the full kit.
The AAP-UX exam weights AX patterns at 30%, and scenario questions live here: "an agent feature failed like X — which pattern addresses it?" Map failure to pattern: silent wrong actions → activity log + provenance; users afraid to enable autonomy → progressive delegation + undo; batch mistakes → plan visibility + sample response; confident wrong answers → verification affordances, not error banners.
Thumbs up/down is where feedback goes to die: sparse, ambiguous, and unactionable. Feedback UX worth building has three properties. It's targeted — "what was wrong?" with a taxonomy (wrong facts / wrong tone / wrong format / refused unnecessarily) that maps one-to-one onto the failure categories engineering's evals track. It's consented and implicit where possible — edits, regenerations, and abandonment are the highest-signal feedback that exists, collected as telemetry with disclosure, not surveillance. And it closes the loop visibly — "you flagged formatting; tables now stay intact," which is the only thing that keeps humans donating labeled data. Co-design the schema with engineering so every signal lands in the eval pipeline; feedback that doesn't reach an eval is a suggestion box bolted to a shredder.
An agent feature without undo is an agent feature users will disable — not because they distrust AI, but because you've made the cost of one mistake unbounded. Reversibility isn't polish; it's the precondition for adoption.
Users grant autonomy to agents the way they grant it to people — incrementally, based on a visible track record, with the power to stop and reverse. Plans before action, logs with provenance, undo by default, gates on the irreversible, and a designed state for every failure: that is what turns agent capability into agent adoption.