AI UX States and the Feedback Loop

Tier: Core · Workflow area: AI features in the client

This playbook gives the AI surface its missing state machine and turns user reactions into engineering signal instead of a decorative thumbs row.

When to use

• •
  Any user-facing AI feature past the demo stage: chat, summaries, suggestions, generated drafts.
• •
  Symptoms of missing states: users staring at silent latency, refusals rendered as generic errors, or hallucination complaints with no capture mechanism.
• •
  You are standing up quality practice (evals) and need real failure cases to seed it.

Skip nothing here for "internal tools" — internal users abandon bad AI surfaces just as fast, they just do it silently.

Prerequisites

• •
  A working AI surface (pb-47-style chat or any generate-and-render flow) built on your design-system components.
• •
  An analytics pipeline that can carry structured events (product analytics or your own event table).
• •
  Somewhere durable to store flagged interactions with their full context — this becomes the eval seed corpus.
• •
  Agreement with whoever owns prompts/models that they will consume this signal; feedback nobody reads is theater.

The workflow

1. 1.

   Implement the AI state rubric as first-class states. Beyond loading/error/empty, an AI surface has states deterministic UIs never needed. Enumerate them in the component's state type — not as ad-hoc conditionals:

   • •
     Thinking — request sent, nothing streaming yet. Show progressive status ("Reading your documents…") when the backend can emit it; a generic shimmer after ~1s; and elapsed-time honesty past ~10s with a cancel affordance. Never a frozen button.
   • •
     Streaming — tokens arriving; stop is visible, input state is defined (pb-47).
   • •
     Uncertainty — the answer may be wrong. Render provenance affordances: citations when grounded (and their absence as a visible "not grounded" cue), confidence framing in copy ("Based on 3 documents…"), and a persistent, non-nagging accuracy disclaimer near generated content.
   • •
     Refusal — the model declined. Style it as guidance, not failure: distinct from error, explains the category of problem, offers a reformulation path. A refusal rendered as a red error toast teaches users the feature is broken.
   • •
     Error — infrastructure failed. Retryable, preserves the user's input, visually distinct from refusal.
   • •
     Empty/cold start — the highest-leverage state: seed prompts and capability examples, because a blank text box transfers the whole discovery burden to the user.
2. 2.

   Build the states as design-system components, once. A GenerationStatus, a Citation chip, a RefusalNotice, a FeedbackBar — reviewed for accessibility (pb-33 covers the screen-reader side of streaming) and reused across every AI surface, so the product's AI states are consistent rather than re-improvised per feature. Add each to your preview/Storybook matrix in all variants.

3. 3.

   Instrument implicit signal — it is bigger and more honest than thumbs. Emit structured events with a shared schema (surface, model, promptVersion — hashed/IDs, not raw text, in analytics):

   • •
     ai_response_regenerated — the strongest cheap dissatisfaction signal.
   • •
     ai_response_stopped mid-stream — wrong direction or too slow.
   • •
     ai_output_edited — for insertable drafts, capture an edit-distance bucket; heavy rewrites mean the draft missed.
   • •
     ai_response_copied / inserted / link-clicked — acceptance signals.
   • •
     ai_session_abandoned — thinking state shown, user left before completion.
4. 4.

   Make explicit feedback cheap, then ask for one more step. Thumbs up/down on every assistant turn, one tap, no modal. On thumbs-down only, offer optional structured tags (wrong, outdated, refused unnecessarily, formatting, unsafe) plus free text. Acknowledge in-UI what feedback does ("helps improve answers") — response rates roughly double when the affordance looks alive rather than decorative.

5. 5.

   Close the loop into the eval set. This is the step that separates signal from theater: flagged interactions (thumbs-down, heavy edits, immediate regenerates) are queued with full context — prompt, retrieved context, output, model and prompt version — into a review tool. A human triages weekly: real failure or user error; genuine failures become eval cases (input plus expected-behavior assertion) in the suite that gates prompt and model changes. Report the loop's health as metrics: feedback rate, triage throughput, and regression count caught by eval cases sourced from user feedback. When a fix ships for a failure class, you have a defensible "we fixed the thing users flagged" narrative — the loop is visible to users and stakeholders alike.

Tool variants

• AI Elements: ships primitives for the transcript chrome (reasoning disclosures, sources, actions like regenerate/copy) — restyle them and add your FeedbackBar; it does not do your analytics or eval plumbing.
• Product analytics (Amplitude/PostHog/GA): fine for the event schema; keep raw prompt/output text out — store those in your own flagged-interactions table keyed by event id.
• Eval tooling (Braintrust, LangSmith, promptfoo, or a homegrown table + CI script): any of them work; the requirement is that user-sourced cases run automatically against prompt/model changes.
• AI-assisted triage: an LLM can pre-cluster the weekly thumbs-down queue by failure type; the accept/reject on each cluster stays human.

Verification

• •
  Every state in the rubric renders from fixtures in the preview matrix, including refusal-vs-error side by side — they must be visually and textually distinct.
• •
  Slow-network test: thinking state escalates through its tiers (status → shimmer → elapsed-time + cancel) instead of sitting frozen.
• •
  Events fire correctly for each behavior (regenerate, stop, edit, copy, thumbs) — verified in the analytics debugger, with no raw text in payloads.
• •
  A thumbs-down travels the whole pipe: appears in the triage queue with full context, becomes an eval case, and that case fails when you deliberately reintroduce the bad behavior.
• •
  Screen reader announces state transitions (thinking → streaming → done) — spot-checked with VoiceOver/NVDA.

Pitfalls

• Refusal styled as error. Users retry infrastructure errors; they reformulate refusals. Conflating them destroys both behaviors.
• Fake precision in uncertainty UI. Rendering "87% confident" from nothing is worse than no signal; show provenance (citations, grounding) — which is real — rather than invented probabilities.
• Thumbs into the void. Feedback collected but never triaged is negative-value: it costs user effort and teaches them the product does not listen.
• Only explicit signal. Thumbs response rates are low single digits; regenerates and edits are the volume signal — instrument them first.
• Raw prompts in analytics. Prompt text is user data; shipping it to a third-party analytics tool creates the pb-50 periphery leak.
• Eval set frozen at launch. If user-sourced failures stop flowing in, the eval suite tests last quarter's product against last quarter's users.

Restricted-environment variant

All of this is your own UI and pipeline — no provider dependency. In restricted shops the constraint is where flagged interactions (which contain prompts and outputs) may be stored: keep the corpus in your tenancy alongside other user data, run triage clustering with an in-tenancy or local model, and keep third-party analytics to text-free event metadata only.

Success criteria

• •
  Every AI surface in the product uses the shared state components — a designer can name the six states and point to each in the app.
• •
  Implicit and explicit feedback flow at measurable rates, and the weekly triage produces eval cases on a steady cadence.
• •
  At least one shipped quality fix per month traces directly to user-flagged failures — and the eval suite prevents its regression.
• •
  Prompt or model changes cannot merge while a user-sourced eval case fails, and stakeholders read the loop's dashboard instead of anecdotes.