Turn DevEx Surveys and Adoption Telemetry into a Platform Roadmap

Tier: Foundation · Workflow area: Developer platform & DevEx

This playbook uses an LLM to turn hundreds of free-text responses plus usage numbers into a defensible, prioritized roadmap, with a traceability audit so every theme you present to leadership can be traced back to real quotes and real numbers, not model invention.

When to use

• •
  Quarterly DevEx survey lands with 100+ free-text responses and you owe leadership a themes-and-priorities readout.
• •
  You are deciding what the platform team builds next and want survey pain + telemetry evidence in one view.
• •
  You are measuring AI coding-assistant rollout (adoption %, retention, satisfaction) and need a narrative, not just charts.

Skip AI for surveys under ~20 responses — read them yourself; the synthesis overhead exceeds the reading time.

Prerequisites

• •
  Survey export as CSV (one row per response, verbatims in named columns) with names/emails removed.
• •
  Usage telemetry exports: portal MAU, golden-path template runs, ticket categories, assistant seat-activation — whatever you have, as CSV.
• •
  Any capable AI assistant with file upload, or an LLM API script. For client or employee data, use your approved enterprise tenant.

The workflow

1. 1.

   Strip identity before anything leaves your machine. Delete name/email/team columns you don't need; replace team names with codes if survey policy promised anonymity. Free-text often self-identifies ("as the payments team's SRE...") — accept that residual risk only inside an approved tenant.

2. 2.

   Synthesize themes with mandatory evidence. Upload the survey CSV:

   Copy

   You are analyzing a developer-experience survey for an internal platform team.
   Attached: survey.csv (columns: role, tenure_band, q1_biggest_friction,
   q2_platform_wish, q3_ai_tooling_feedback).
   
   Produce:
   1. 5-9 themes, each with: name, one-paragraph description, estimated share of
      respondents mentioning it, and 3 VERBATIM quotes (copied exactly, with row
      numbers).
   2. A "contradictions" section: places where respondents disagree.
   3. A "long tail" list: real issues mentioned by only 1-2 respondents.
   Do not invent quotes. If fewer than 3 quotes support a theme, say so and
   demote it.
   

3. 3.

   Audit the quotes. Spot-check at least one quote per theme against the CSV by row number (grep or Ctrl+F). A quote that doesn't exist verbatim kills the theme until you re-verify it manually. This is the claim-audit discipline from the common core applied to survey data.

4. 4.

   Join with telemetry. Upload the usage CSVs in the same conversation:

   Copy

   Now cross-reference themes against the attached telemetry. For each theme,
   state whether usage data supports, contradicts, or is silent on it — citing
   the specific metric and value. Flag any theme where loud survey sentiment
   contradicts quiet usage data (or vice versa) as "investigate before acting."
   

5. 5.

   Prioritize with an explicit model. Ask for a RICE (or effort/impact) table over the supported themes, with the model's reasoning per score shown. You then adjust the scores — the LLM proposes, you decide. Scores you can't defend in a roadmap review get recomputed by hand.

6. 6.

   Draft the readout, keep authorship. Generate a one-page readout (themes, evidence, top 3 investments, explicitly deferred items). Rewrite the recommendations section in your own words — the priorities are your call and your accountability.

Tool variants

• Chat assistant with file upload: simplest path; handles a few thousand rows comfortably. Use code-execution/analysis mode for the telemetry joins so counts are computed, not guessed.
• LLM API script: for recurring surveys, script the synthesis with a fixed prompt and structured-output schema (themes[].quotes[].row) so quarter-over-quarter outputs are comparable.
• DX/GetDX-class platforms: ship built-in AI summaries — still run the quote audit on their output before forwarding; vendor summaries hallucinate too.

Verification

• •
  Every theme has ≥3 quotes verified verbatim at the stated row numbers.
• •
  Respondent-share estimates sanity-checked: sum of theme shares can exceed 100% (multi-mention) but no single theme claims more mentions than responses exist.
• •
  Every telemetry claim cites a metric you can re-derive from the raw CSV.
• •
  Themes flagged "investigate" are excluded from the priority list until investigated.

Pitfalls

• Counting by vibes. LLMs estimate frequencies badly from prose. If the exact share matters, have code execution count keyword-tagged rows, or count yourself.
• Squeaky-wheel bias. Ten angry verbatims from one team ≠ an org-wide theme — that's why the telemetry cross-check step exists.
• Losing the long tail. The single security-relevant response matters more than the tenth "CI is slow." Always request and read the long-tail list.
• Letting the model set priorities. RICE scores from an LLM are a starting grid, not a decision. It knows nothing about your headcount or politics.

Restricted-environment variant

Employee survey data is personal data in most jurisdictions and often covered by works-council or HR policy:

• •
  Use only your organization's approved enterprise AI tenant with training-on-data disabled; never a personal consumer account.
• •
  Pre-aggregate where policy demands: run a local script to bucket verbatims by keyword first, and send only the bucketed, de-identified text.
• •
  Services companies: client-satisfaction verbatims naming client staff or projects must be redacted (CLIENT_A, PROJECT_X) before synthesis — same rule as client telemetry.

Success criteria

• •
  A readout produced in under 2 hours that previously took a day or more.
• •
  Every theme in the readout survives a row-number spot-check by a skeptical colleague.
• •
  At least one loud-survey/quiet-telemetry contradiction caught before it became a roadmap item.
• •
  The final priority list reflects your judgment, with AI-proposed scores visibly adjusted where you disagreed.