Mine Tickets, NPS and Reviews into Counted Themes

Workflow area: Discovery & Research · Tier: Core · Hands-on time: ~35 minutes

The difference between "users complain about search" and "search relevance is theme #2 at 14% of tickets, up from 9% last quarter" is the difference between an anecdote and an argument. LLMs made the classification step cheap. What they did not make cheap is trust: a model will happily sort ten thousand items into whatever themes it invents, drift its definitions halfway through, and hand you confident percentages built on mush. The recipe fixes the schema first, batches the classification, and audits a sample per theme before any number leaves your desk.

When to use — and when not to

• Use it quarterly on the full feedback corpustickets, NPS verbatims, app-store reviews, sales-call complaint snippets — to rank problem areas with counts before planning.
• Use it when someone claims "everyone is asking for X"a counted classification either confirms the anecdote or shrinks it to the four loud customers it actually is.
• Use it for before/after readssame schema run on pre-launch and post-launch windows shows whether a fix actually moved its theme's share.
• Skip it for corpora under ~100 itemsjust read them. An afternoon with 60 verbatims builds intuition no classification output gives you, and the counts would be noise anyway.
• Skip fully automated pipelines that publish counts nobody auditeda dashboard of unaudited AI classifications is the analytics equivalent of a hallucinated quote, with better fonts.

The recipe

1. 1.

   Fix the schema before the model sees the data. Read 50 random items yourself and draft 8–12 themes with one-line definitions and an example each. Include other and not-feedback (spam, thanks, duplicates). If you let the model invent themes per batch, batch 1's "onboarding friction" and batch 7's "activation issues" will silently be the same theme counted twice.

2. 2.

   Classify in batches with a structured-output prompt. 100–200 items per batch, each with a stable ID:

   Copy

   You are classifying customer feedback into a FIXED schema. Use ONLY
   these themes — do not invent, merge, or rename themes:
   
   [PASTE SCHEMA: id, name, one-line definition, example]
   
   For each item, output one line:
   ITEM_ID | THEME_ID | SEVERITY (blocker/major/minor) | CONFIDENCE
   (high/low) | QUOTE (verbatim fragment ≤15 words that justifies
   the theme)
   
   Rules:
   - An item that fits nothing gets "other" — never force-fit.
   - An item raising two themes gets its PRIMARY theme only.
   - The QUOTE must be copied verbatim from the item, not paraphrased.
   - Mark CONFIDENCE low whenever you hesitated.
   
   Items:
   [PASTE BATCH WITH IDs]
   

3. 3.

   Aggregate in a sheet, not in the chat. Paste every batch's output into Google Sheets, pivot on THEME_ID for counts and shares, and keep the item-level rows — they're your audit trail. Never ask the model to total its own counts across batches; arithmetic across a long context is exactly where it slips. A healthy pivot:

   Copy

   THEME                    COUNT   SHARE   LOW-CONF
   billing-confusion          412    13.7%      31
   search-relevance           389    12.9%      18
   mobile-sync-failures       267     8.9%      52
   other                      488    16.2%       —
   

4. 4.

   Run the 20-item audit per theme. For each theme you plan to cite, pull 20 random items classified into it and read them yourself. Count the misclassifications: 19–20 correct means the count is quotable; 16–18 means quote it as approximate; below 16 means the theme's definition is broken — rewrite it and reclassify. Audit other too: a fat other hides your next theme.

5. 5.

   Write the findings memo with counts, quotes, and caveats. One page: top themes with share and quarter-over-quarter movement, two verbatim quotes per theme (pulled from the audit, so you've verified them), the audit precision per theme, and what the corpus can't tell you — silent churners don't file tickets, and NPS verbatims over-represent extremes.

Make it yours

• Dovetail as the system of record: push classified items into Dovetail with theme tags so the counted themes and pb-02's interview themes live in one searchable repo, and pb-20's RAG can cite both.
• Severity-weighted ranking: a theme that's 6% of volume but 80% blockers outranks a 12% theme of minor gripes — add a weighted score column to the pivot before you rank.
• The monthly delta run: after the first full pass, classify only the new month's items with the same schema and track share movement — theme trend lines are more decision-useful than any single snapshot.

Quality check

• The schema was fixed before classificationthemes invented mid-run make every count incomparable; if you changed the schema, you reclassified everything.
• Every quoted number survived its 20-item auditand the memo states the audit precision next to the count, because "89% precise" is an honest number and stakeholders can handle it.
• Quotes are verbatimspot-check the memo's quotes against source items; classification prompts paraphrase under pressure exactly like extraction prompts do.
• other is under ~20% and was reada bloated other means the schema is missing a theme, not that users are incoherent.
• The denominator is stated"14% of Q3 tickets" not "14% of users"; ticket share and user share are different claims and the model will conflate them if you let it.