Define the AI Feature Metrics Ladder

Workflow area: Evals & AI Quality · Tier: Advanced · Hands-on time: ~40 minutes

Most AI features launch with two numbers: a benchmark score and a usage count. Both flatter. Usage counts satisfaction and frustration identically — a user regenerating five times looks "engaged." The ladder replaces vanity metrics with a chain that runs from model behavior to business outcome, and it forces the distinction dashboards love to blur: a support bot that contains a ticket (user got their answer) and one that deflects it (user gave up) produce the same "no escalation" event.

When to use — and when not to

• Use it before instrumentation is builtthe spec must precede the launch, because retrofitting events after GA means a quarter of unmeasurable performance.
• Use it when someone calls the feature "successful" from usage alonethe ladder is the antidote to "10k generations this week" storytelling.
• Use it for any feature where the AI output can be silently wrongsummaries, drafts, answers; anywhere users can accept bad output without telling you.
• Skip the full ladder for internal tooling with expert usersa weekly thumbs-up review and a complaint channel may be proportionate; instrument the top rung only.
• Skip it if you can't change the product to add signalsa ladder without acceptance affordances (edit, accept, regenerate, escalate) measures nothing; ship the affordances first.

The recipe

1. 1.

   Define "successful outcome" in one sentence. Not "user got a response" — the user's job was done: ticket resolved without reopening, draft sent with under 20% edits, answer accepted without a follow-up search. Everything on the ladder hangs off this sentence, so write it with your engineer and designer in the room.

2. 2.

   Generate the ladder draft with AI, rung by rung. Paste into Claude:

   Copy

   Build a metrics ladder for an AI feature, from model behavior up to
   business outcome.
   Feature: [WHAT IT DOES AND FOR WHOM]
   Successful outcome definition: [YOUR ONE SENTENCE]
   User affordances that exist: [e.g. accept / edit / regenerate /
   thumbs-down / escalate to human]
   
   Produce 5 rungs, bottom to top:
   R1 MODEL QUALITY (offline eval metrics)
   R2 INTERACTION SIGNALS (acceptance rate, edit distance,
      regeneration rate, abandonment mid-flow)
   R3 TASK OUTCOME (containment vs deflection — define BOTH and how
      we tell them apart; complaint rate; reopen rate)
   R4 RETENTION (repeat usage by the same user on the same job)
   R5 BUSINESS (cost per successful outcome, margin per user)
   
   For each rung: metric definition, the exact event(s) needed to
   compute it, the failure mode of reading this metric alone, and a
   target-setting note. Be specific about how deflection is detected —
   e.g. user exits then contacts support within 24h, or repeats the
   query in another channel.
   

   Sample output for the rung most teams get wrong:

   Copy

   R3 — CONTAINMENT vs DEFLECTION
   Containment = session ends after bot answer AND no support contact
   from same account within 24h AND no repeat of same intent within 7d.
   Deflection = session ends after bot answer BUT support contact or
   intent-repeat follows. Metric: containment / (containment +
   deflection + escalation). Events needed: session_end{intent_id},
   support_contact{account, ts}, intent_repeat{intent_id, ts}.
   Failure mode if read alone: raising the friction to reach a human
   improves "containment" while destroying trust.
   

3. 3.

   Turn rung two and three into an instrumentation spec. One table: event name, properties, trigger condition, owner, destination (Amplitude for product analytics, Langfuse for per-trace signals like regeneration chained to a specific prompt version). Regeneration deserves its own care — log what changed between attempts, because "regenerated with the same input" means quality failure while "regenerated after editing input" means exploration.

4. 4.

   Set targets asymmetrically. For each metric decide: target, guardrail (the value at which you page the team), and review cadence. Acceptance targets come from a two-week baseline, not from wishes. Complaint and deflection rates get guardrails immediately even before targets exist — you always know what "alarming" is.

5. 5.

   Wire the top rung to the bottom. Cost per successful outcome = (tokens plus infra per session × sessions) ÷ successful outcomes. This single number connects your Langfuse traces to your CFO conversation, and it's the metric that pb-46's unit-economics model consumes. Put it on the same dashboard as acceptance so nobody optimizes one against the other invisibly.

Make it yours

• Braintrust for the R1–R2 join: log production traces with acceptance outcomes attached, then promote real rejected outputs into your eval set weekly — the ladder becomes a flywheel where live failures grow the offline suite.
• Support-bot variant: add a sixth signal — sentiment at escalation. A user arriving at a human agent already angry is a deflection cost your containment number hides; a one-field CSAT on escalated chats captures it.
• Copilot/drafting variant: replace containment with edit distance and time-to-send; acceptance of unedited output is your top interaction signal, and a rising accept-without-reading rate is a risk metric, not a win.

Quality check

• Deflection is operationally definedif you can't state the exact event sequence that distinguishes it from containment, the ladder's key rung is fiction.
• Every metric has its named failure modea ladder that doesn't say how each metric lies will be gamed within a quarter.
• Regeneration is diagnosed, not just countedsame-input regeneration and changed-input regeneration are logged distinctly.
• Cost sits next to qualitycost per successful outcome shares a dashboard with acceptance rate, so trade-offs are visible.
• Events exist in the tracker before launch reviewa spec is done when engineering has ticketed every event, not when the doc is written.