Design Feedback That Feeds the Eval Pipeline

Workflow area: AI-UX Evaluation · Tier: Advanced · Hands-on time: ~45 minutes

Most feedback UI fails twice. It fails users, who click a thumb, watch nothing change, and correctly stop clicking. And it fails the eval pipeline, because "thumbs down" doesn't say whether the answer was wrong, rude, slow, or just too long. Meanwhile the evals that decide what ships get written by engineers grading accuracy — while tone, appropriateness of hedging and refusal quality go ungraded because nobody wrote the rubric. Both failures are design work: the capture taxonomy is an interaction design problem, and the rubric is a quality-definition problem designers are trained for. PAIR's feedback-and-control guidance frames the principles; this playbook builds the pipeline.

When to use — and when not to

• Use it when your feedback data is unusableif the model team says "the thumbs data doesn't tell us anything," the fix is capture design, not more volume.
• Use it when feedback rates have collapsedsustained sub-1% engagement usually means users learned that feedback is a black hole; loop-closing is the treatment.
• Use it when evals grade only accuracyif no eval fails for a right answer delivered with the wrong tone, hedge or refusal style, design has no seat yet; write the rubric.
• Skip heavy taxonomies on low-traffic surfacesa feature with fifty weekly users needs interviews, not instrumented feedback categories.
• Skip implicit-signal work if you can't get consent infrastructurebehavioral signals without disclosure are surveillance; if the org can't do it right, do explicit-only.

The recipe

1. 1.

   Replace the thumb with a two-tap taxonomy. Tap one stays cheap: this helped / this didn't. Tap two, only on negative, asks what kind of failure — and the options come from your pb-48 failure-mode map, not a generic list: "factually wrong," "outdated," "missed part of my question," "wrong tone," "too long/short," "refused unnecessarily." Six options maximum, user language, one tap each, free text optional and never required. Every category must route somewhere: wrong → eval set candidate, outdated → retrieval index review, tone → voice-guide regression. A category with no consumer gets cut.

2. 2.

   Design implicit signals as a consented tier. Behavior often outranks button clicks as signal: regenerated then abandoned (failure), copied to clipboard (success), edited-heavily-then-used (partial), stopped mid-generation (mismatch). Pick the three to five signals that map cleanly to quality, and disclose them where the choice happens — a settings item in plain language: "Help improve suggestions by sharing how you use responses (edits, copies, regenerations). Never the content of your documents." Default per your org's privacy posture, off-switch always honored, and the signal list visible, not buried in a policy PDF.

3. 3.

   Close the loop visibly, at three altitudes. Feedback UI earns future feedback by showing consequences: immediate — the acknowledgment does something ("Thanks — we'll deprioritize answers like this for you") when true; per-user — a "your feedback" view showing what they flagged and any resulting change; product-level — release notes that credit the loop ("You told us summaries buried the numbers — tables now lead"). The third is the strongest lever and costs one paragraph per release.

4. 4.

   Write the UX-graded eval rubric — design's seat at the table. Take the transcripts and outputs the model team already evaluates, and add the dimensions accuracy grading misses. Draft it with AI, then calibrate it yourself:

   Copy

   You are helping a designer write eval rubrics for an AI feature's
   output quality, beyond factual accuracy.
   
   Feature: [DESCRIPTION + voice principles + failure-mode map]
   
   Draft a rubric with 5-7 UX dimensions, e.g.: tone fidelity to our
   voice, hedge calibration (hedged when uncertain, confident when
   grounded), refusal quality (clear, unmoralizing, offers
   alternative), completeness (addressed every part of the ask),
   scannability (structure matches content type), personalization
   appropriateness (used context without creepiness).
   
   For EACH dimension: a 1-5 scale where 1, 3 and 5 are defined by
   CONCRETE, checkable descriptions (not "poor/okay/great"), plus
   one PASS example and one FAIL example written out.
   The rubric must be usable by a grader who is not a designer, and
   by an LLM judge given the same text.
   

   Sample of the precision to demand:

   Copy

   HEDGE CALIBRATION
   5: Every uncertain claim carries a specific hedge naming the
      reason ("source last updated 2023"); grounded claims are
      stated plainly with no hedge.
   3: Hedges present but generic ("this may not be accurate").
   1: Uniform hedging on everything, or zero hedging including on
      claims the context cannot support.
   

5. 5.

   Wire capture to rubric and pilot for two weeks. Feedback categories and rubric dimensions should share names — a "wrong tone" tap and the tone-fidelity dimension are the same concept measured from two directions, which is what makes design's metrics legible to the model team. Run the loop manually first: grade twenty real outputs with the rubric in Google Sheets (one row per output, one column per dimension), check inter-rater agreement with a colleague on five of them, tighten any dimension where you disagreed by more than a point, and only then hand it over for LLM-judge automation.

Make it yours

• Google Sheets as the eval bench: keep the manual grading sheet alive even after automation — ten hand-graded outputs a week is the calibration set that tells you when the LLM judge drifts.
• Claude as disagreement analyst: paste pairs of outputs where you and a colleague scored differently and ask what ambiguity in the rubric wording allowed the split — then patch the wording, not the score.
• Support-channel mining: run the taxonomy backward over support tickets mentioning the AI feature — tickets are feedback from users who never touch feedback UI, and they weight your categories by real pain.

Quality check

• Every feedback category routes to a named consumera team, a pipeline, an eval set; orphan categories were cut.
• Negative feedback resolves to a failure modethe taxonomy speaks pb-48's language, so signal joins directly to the failure map.
• Implicit signals are disclosed in plain languagea user can list what's collected without reading a policy; the off-switch works.
• The loop closes visibly at least at product levelsome released change is publicly credited to user feedback each cycle.
• Rubric anchors are checkabletwo graders scored five outputs within one point on every dimension before automation was allowed.
• Design metrics appear in the eval dashboardtone, hedging and refusal quality fail builds the same way accuracy does.