Mine Feedback at Scale With Precision Audits

Workflow area: Research & Synthesis · Tier: Core · Hands-on time: ~40 minutes

Feedback mining is the highest-leverage, highest-risk research task AI touches. Highest leverage because no human will ever read all the verbatims, and the themes genuinely are in there. Highest risk because the output is a table of counts — and counts look like facts. A theme labeled "sync failures: 412 mentions" will move a roadmap whether or not the classifier actually put sync complaints in that bucket. The precision audit is the difference between quantified insight and confident noise.

When to use — and when not to

• Use it when verbatims outnumber your reading capacitya thousand-plus items across reviews, tickets, NPS comments or survey free-text is the natural habitat.
• Use it before roadmap and OKR seasonthemes with audited counts are the strongest evidence a designer can bring to a prioritization fight.
• Use it on a recurring cadencethe same schema run monthly turns feedback into a trendline; a theme growing 40% quarter over quarter is a finding no single ticket contains.
• Skip it for small batchesunder a couple hundred items, read them yourself; you will catch nuance no schema holds and the audit overhead outweighs the savings.
• Skip it as a substitute for researchfeedback mining tells you what vocal users report, not what silent users experience; it aims research, it does not replace it.

The recipe

1. 1.

   Assemble and de-duplicate the corpus. Export reviews, tickets and NPS verbatims into one file with columns: id, source, date, text. Strip obvious duplicates and canned agent replies. Note the date range — counts without a time window are meaningless.

2. 2.

   Draft the theme schema from a sample, not from your head. Paste 100 random items into Claude and ask for candidate themes. Then edit: merge overlaps, cut themes you cannot act on, add ones you know exist. Ten to fifteen themes plus "other" is the workable range.

3. 3.

   Classify with a structured-output prompt. Run the corpus in batches:

   Copy

   Classify each feedback item into the schema below. Output JSON lines,
   one per item: {"id": ..., "theme": ..., "confidence": "high|low",
   "evidence_phrase": "verbatim words that triggered the label"}
   
   Themes:
   - sync_failure: data not syncing, lost edits, conflict errors
   - pricing_value: cost complaints, plan confusion, "not worth it"
   - onboarding_confusion: lost in first session, unclear setup
   - performance: slow load, lag, crashes
   [... YOUR REMAINING THEMES WITH ONE-LINE DEFINITIONS ...]
   - other: fits no theme above
   
   Rules: exactly one theme per item — choose the dominant complaint.
   The evidence_phrase must be copied verbatim from the item.
   Mark confidence "low" whenever the fit is arguable. Never leave
   "other" empty just to look decisive.
   
   Items: [PASTE BATCH]
   

4. 4.

   Run the precision audit before believing any count. For each theme, pull ~20 random classified items (all of them if a theme has fewer) and judge each label yourself: correct or wrong. Twenty is enough to catch a broken bucket — a theme at 70% precision is misdirecting nearly a third of its count. Record per-theme precision:

   Copy

   THEME                 COUNT  AUDITED  CORRECT  PRECISION
   sync_failure            412       20       19       95%
   pricing_value           388       20       14       70%   <- fix or flag
   onboarding_confusion    203       20       18       90%
   

   For any theme under ~85%, read the misclassified items, tighten that theme's definition, and re-run the classification for that theme. If "other" exceeds 15% of the corpus, your schema is missing a real theme — mine "other" for it.

5. 5.

   Write the roadmap brief with precision attached. One page: theme counts with date range, per-theme precision, three verbatim quotes per top theme (pulled from audited-correct items only), and what each theme suggests investigating — not what to build; the verbatims describe symptoms, and prescribing fixes from symptoms is how teams ship the wrong thing confidently.

Make it yours

• Sprig or in-product surveys as the fresh stream: wire new survey free-text into the same schema so classification runs continuously; audit a smaller rolling sample monthly instead of a big-bang audit.
• Dovetail as the system of record: import the classified items as tagged data so every theme count is clickable through to its verbatims — stakeholders trust counts they can drill into.
• Claude Project for schema versioning: keep the schema, per-theme precision history and definition changes in one Project; when precision jumps after a definition edit, you have learned something durable about the category.

Quality check

• Every count has a date range and a precision figure next to ita naked count is a rumor with formatting.
• The audit sample was randomcherry-picking obvious items inflates precision; use a random draw per theme.
• Evidence phrases are verbatimif the model paraphrased the trigger words, it may have classified its own paraphrase, not the user's complaint.
• "Other" was mined, not ignoreda fat "other" bucket is your next theme hiding in plain sight.
• Quotes in the brief come from audited-correct itemsquoting an unaudited item under an audited count quietly breaks the whole chain of evidence.