Measure Deflection Honestly and Run the Review Loop

Tier: Core · Workflow area: Governance & measurement · ~40 minutes

This playbook builds the measurement workbook and the operating rhythm behind UC-45 and UC-50. It is the desk's defense in both directions: against overclaiming to management, and against a quietly degrading bot that the dashboard says is fine.

When to use

• The bot has been live 30+ daysenough conversations to measure without anecdote. Before that, review every conversation instead; you can.
• A renewal or expansion decision is comingthe vendor's deflection slide will anchor the price. Walk in with your strict number and the reconciliation table.
• Someone quoted the dashboard in a leadership deckthe moment an unaudited number becomes an official number, this playbook is overdue.

When NOT to use

• To manufacture a success storyif the honest number is 8% while the dashboard says 46%, the honest number ships. The whole point is a figure you can defend under hostile questioning.
• On volumes too small to sliceunder ~200 conversations a month, weekly percentages will swing wildly; report monthly and lean harder on the conversation review.
• As a substitute for fixing coveragemeasurement finds problems; pb-42 and pb-40 fix them. A beautiful workbook over a broken bot is instrumentation of failure.

The recipe

1. 1.

   Write the strict definition and pin it to the workbook. A conversation counts as deflected only if all four hold: the bot delivered an answer or completed an action (not a link dump into silence); the user did not escalate in that conversation; no ticket from the same user on the same topic within 7 days; and — where feedback exists — the user did not rate it negative. Everything else is escalated, abandoned (user left mid-flow before an answer), or unresolved-repeat (came back within 7 days). Four buckets, no overlap.

2. 2.

   Instrument the four core metrics. Strict deflection rate; abandonment rate; reopen/recontact rate (the 7-day comeback); BSAT (post-chat rating, reported alongside its response rate — a 90% BSAT on 4% response is decoration). Pull raw conversation and ticket data via API rather than trusting rollups:

   Copy

   # Recontact check: tickets opened within 7 days by users the bot "deflected"
   curl -s -u "$SN_USER:$SN_PASS" \
     "https://yourinstance.service-now.com/api/now/table/incident?sysparm_query=opened_atRELATIVEGT%40dayofweek%40ago%407%5Ecaller_idIN$DEFLECTED_USER_IDS&sysparm_fields=number,caller_id,short_description,opened_at" \
     | jq '.result | length'
   

   Matching topic is fuzzy; match on same user + same category first, and let the weekly review adjudicate the ambiguous ones.

3. 3.

   Build the reconciliation tab. Three columns per month: vendor-dashboard deflection, strict deflection, and the explained gap. The gap decomposes into named causes — "vendor counts link-clicks as resolutions," "vendor excludes abandoned chats from the denominator," "7-day recontacts not tracked." Getting the vendor's counting rules in writing is half the exercise; put the email in the workbook.

4. 4.

   Run the weekly conversation review — 30 minutes, sampled, graded. Pull 20 conversations: 10 random, 5 abandoned, 5 escalated. Two people grade independently against a fixed rubric, then compare:

   Copy

   Grade this support-bot conversation on four axes, 1-3 each:
   ANSWER   3 correct+grounded / 2 partially right / 1 wrong or invented
   EXIT     3 clean resolution or warm handoff / 2 clumsy exit / 1 dead end
   TONE     3 professional / 2 awkward / 1 frustrating
   MISS     name the single biggest improvement, in one line
   
   Transcript: [paste]
   

   Use the LLM as a third grader if you like, but never as the only one — a model grading a model inherits the model's blind spots.

5. 5.

   Turn review findings into a ranked fix list. Every grading session ends with items tagged KB gap (route to article work), threshold (route to pb-40 tuning), intent gap (route to pb-42 backlog), or matrix (route to pb-43 review). An observation without a route is a complaint; the tag is what makes the loop a loop.

6. 6.

   Publish one honest page monthly. Strict deflection with trend, abandonment, recontact rate, BSAT with response rate, tickets-per-user trend (the denominator check that catches deflection "wins" caused by users bypassing the bot entirely), plus the vendor-vs-strict gap and the top three fixes shipped from review. One page, same layout every month — pb-48's narrative recipe turns it into prose.

7. 7.

   Compare against the pb-42 ceiling quarterly. Your portfolio math predicted a maximum. If strict deflection is far under it, the review tags tell you which lever is failing; if it is somehow above it, re-check the strict definition before celebrating — inflation found a way in.

Make it yours

• ServiceNow shopbuild the workbook from Virtual Agent conversation tables and incident data via the Table API; Performance Analytics can render the trend, but the strict-definition computation stays in your export, where the logic is inspectable.
• Freshservice shopexport Freddy conversation analytics and reconcile against ticket creation; Freddy's own deflection figure goes in the vendor column of the reconciliation tab, never in the headline.
• Custom-bot shopyou own the logs, so instrument at the source: emit a structured end-of-conversation event (outcome, intent, escalated, rating) and the workbook becomes one query instead of a stitching exercise.

Quality check

• The denominator is explicit. Every rate names what it divides by, in the workbook, in writing. Most measurement fraud is denominator fraud.
• Samples are random, plus targeted. The 10 random conversations are drawn by script, not scrolled-and-picked; the targeted abandoned/escalated five are where the lessons live.
• Two graders, tracked disagreement. If graders disagree on more than a quarter of rows, the rubric is ambiguous — fix the rubric, not the graders.
• Numbers survive re-derivation. Once a quarter, have someone else recompute strict deflection from raw data. If they cannot reproduce your figure, the workbook has drifted.