Classify a thousand app-store reviews into themes with a structured-output schema, then sample-audit roughly twenty items per theme for precision before a single claim reaches the roadmap brief. The seed data below contains classification traps — sarcasm, wrong-app reviews, near-duplicates — designed to inflate exactly the theme your stakeholders most want to hear.
Sana is the product designer for Trailhawk, a fictional hiking-navigation app. Leadership believes offline maps are the top complaint and wants budget assigned accordingly; Sana has 1,000 unread reviews and a suspicion that "what leadership believes" and "what the reviews say" are drifting apart. AI classification makes reading all 1,000 feasible in an afternoon — but an unaudited classifier is just leadership's bias with extra steps. The discipline in this lab is the precision audit: no theme count gets quoted until you have hand-checked a sample of what the model filed under it.
Thirty seed reviews. You will expand these to a 1,000-row synthetic dataset in step 2, but the traps live here — so this block is also your audit ground truth. Paste-ready.
R01 ★1 Battery drain is unreal. 20% per hour with the screen off.
R02 ★5 Offline maps saved my life on the Annapurna circuit. Flawless.
R03 ★2 Oh sure, the offline maps "work" — if you enjoy a grey grid
where the trail should be. Downloaded three times. Nothing.
R04 ★1 Crashes every time I rotate my phone on the route screen.
R05 ★4 Love it, but battery usage on long hikes is scary.
R06 ★1 Since the update it logs me out every single day. Every day!
R07 ★5 Great game, my kids love collecting the little hawk badges!
Best tower defense I've played this year.
R08 ★2 The new subscription popup appears EVERY launch. I paid
already. Stop.
R09 ★1 Offline map downloaded fine, then vanished at the trailhead.
No signal, no map, 12km from the car park.
R10 ★3 Decent, but elevation profile is wildly wrong on switchbacks.
R11 ★1 App deleted my recorded route after 6 hours of hiking. Livid.
R12 ★5 Perfect. The offline maps alone are worth the subscription.
R13 ★2 Battery died 3 hours in. My old Garmin lasted 3 days. Just
saying.
R14 ★1 Crashes on launch since yesterday's update. Pixel 8.
R15 ★2 Wonderful, another update, another crash. Truly innovative
ways to fall over, guys. ★★ for creativity.
R16 ★1 Logged out mid-hike and it wanted my password. On a mountain.
With no signal. Think about that.
R17 ★4 Would be 5 stars if the maps worked offline like advertised.
R18 ★1 The route-sharing link sends my friends to a blank page.
R19 ★3 Elevation gain says 400m, my watch and the park sign say 650m.
R20 ★1 Offline maps: downloaded, verified, opened at trailhead, grey
tiles. Fourth trip in a row. I'm done.
R21 ★5 Downloaded maps for all of Snowdonia, zero signal all day,
navigation flawless. This is the killer feature.
R22 ★2 App is fine but it drained 40% battery on a 2-hour walk.
R23 ★1 Crashes when I rotate the phone on the route screen.
R24 ★1 Why do I need to log in again every morning? Fix your tokens.
R25 ★3 The hawk badge animations are cute but I'd trade them all
for a battery saver mode.
R26 ★1 Route deleted itself at the summit. Six hours. Gone.
R27 ★5 Customer support sorted my subscription in ten minutes. Rare!
R28 ★2 Elevation data off by 30% again. Planning around it is
impossible.
R29 ★1 Crashes every time I rotate my phone on the route screen!!
R30 ★4 Nearly perfect — just stop logging me out weekly.
Skim the seeds and write your own theme guess — five minutes, no AI. Predict the top three themes and roughly what share of reviews each covers. This is the anchor you will compare the classifier against.
Generate the full synthetic dataset. Prompt: "Generate 970 additional synthetic app-store reviews for a hiking-navigation app, one per line, numbered R31–R1000, in the same style and rough topic proportions as these 30 seed reviews [paste seeds]. Keep each under 30 words with a star rating." Generate in batches if the model truncates. Paste everything into your spreadsheet. (In real work this step is your actual export; everything after this is identical.)
Define the schema before classifying. With your AI, agree a fixed label set — for example: offline-maps, battery, crashes, auth/login, data-loss, accuracy, monetization, sharing, praise-general, support, other/invalid. Write a one-line inclusion rule per label. Rule of thumb: if two labels need a paragraph to tell apart, merge them.
Classify with structured output. Prompt: "Classify each review. Return one line per review: ID | label from this exact list | confidence high/med/low | 5-word evidence phrase quoted from the review. Use other/invalid for reviews not about this app. Never invent a label." Run in batches of ~100. Spot that evidence phrase column — it forces the model to point at words, which makes your audit tractable.
Tally, but do not report. Pivot the labels into counts. Note which theme leads. Now stop: these numbers are unaudited and quoting them is the exact failure this lab exists to prevent.
Run the precision audit. For each of your top five themes, sample up to 20 classified reviews (use every seed row that landed there, plus random generated rows). Hand-check each: does this review genuinely belong? Record precision per theme — correct / sampled. Watch specifically for: sarcastic praise counted as praise, reviews for a different app counted at all, and near-duplicate reviews inflating a theme's count.
Correct the counts and re-tally. Apply what you found: discount themes by their measured error rate, deduplicate the near-identical rows, and reroute the misfiled seeds. Note whether the leading theme changed. Log every seed-row correction — that list is checked against the answer key.
Write the roadmap-input brief. One page: top themes with audited counts and precision scores, two verbatim quotes per theme (real ones — pull from the seeds), a "data quality" paragraph stating sample sizes and known limits, and one recommended next action per theme. Any theme you did not audit gets listed under "unaudited — directional only."
Precision per theme is the number that makes stakeholders trust the chart — "offline maps: 41 mentions, 95% audited precision" survives a boardroom challenge; "the AI said 41" does not.
Classification at scale is cheap; trustworthy classification is classification plus a sampled precision audit per theme. The audit costs twenty minutes and is the entire difference between mining feedback and laundering your stakeholders' assumptions through a model.