Critique User Stories for Testability Before the Sprint Starts

Tier: Foundation · Workflow area: Test design & planning

QA's oldest superpower is asking "what exactly does this mean?" before anyone writes code. This playbook industrializes that instinct: an LLM reads each story hunting ambiguity, missing acceptance criteria, contradictions, and unmeasurable claims — and you carry the defensible findings into refinement as questions, not accusations.

When to use

• Backlog refinement prepthe night before grooming, run the sprint candidates through the critique and arrive with sharp questions.
• Stories inherited from another team or a clientyou lack the hallway context, so written gaps hurt you most.
• A recurring pattern of "works as designed" bug closuresthe symptom that requirements, not code, are your defect source.

When NOT to use:

• Well-worn story templates with strong Example Mapping already donethe critique will restate what the examples settled.
• Spike or research ticketsthey are supposed to be open-ended; testability critique produces noise.
• As a substitute for talking to the product owner. The output is your question list for the conversation, never a replacement for it.

Prerequisites

• Backlog export accessJira/Linear filter export or API token for the sprint candidates; jq locally if you are trimming JSON.
• An approved assistant under your data policy — story text often contains client and roadmap information, so check what may be pasted where before you start.
• Your Definition of Ready (or the nearest thing to one) written down, so the critique scores against your team's bar.
• A refinement session on the calendar within a day or two — findings go stale fast once development starts.

The recipe

1. 1.

   Export the sprint candidates. Pull story summaries, descriptions, and acceptance criteria. From Jira Cloud, a filter export to CSV works; strip fields you don't need and any client-identifying data:

   Copy

   jq -r '.issues[] | "## " + .key + ": " + .fields.summary + "\n" + (.fields.description // "NO DESCRIPTION")' \
     backlog-export.json > stories.md
   

2. 2.

   Run the adversarial critique, one structured pass. The framing matters: the model must attack, not summarize.

   Copy

   You are a hostile-but-fair test lead reviewing user stories for TESTABILITY.
   For each story below, report:
   1. AMBIGUOUS TERMS — words with multiple readings ("fast", "recent",
      "appropriate", "user"), each with two conflicting interpretations.
   2. MISSING ACs — behavior implied but never specified (error paths, empty
      states, limits, permissions, concurrency).
   3. CONTRADICTIONS — ACs that conflict with each other or with another story.
   4. UNMEASURABLE CLAIMS — outcomes with no observable pass/fail signal.
   5. TESTABILITY SCORE — 1-5 with one-line justification.
   Do NOT propose solutions. Do NOT invent requirements. Questions only.
   
   [paste stories.md]
   

3. 3.

   Verify each finding against the source. Models over-flag: some "ambiguities" are settled by team convention or a linked design. Mark each finding valid, known-context (settled, but worth writing down), or noise. A useful pass looks like:

   Copy

   PROJ-214 "Notify users of failed payments"
   AMBIGUOUS: "notify" — in-app banner or email? "users" — the payer only,
   or all org admins?
   MISSING AC: retry behavior; what happens if notification itself fails.
   CONTRADICTION: AC-3 says "immediately"; PROJ-198 batches payment events
   at 15-minute intervals.
   SCORE: 2/5 — no observable definition of delivered notification.
   

   That contradiction against a different story is the class of catch humans miss and models excel at — verify it by opening PROJ-198.

4. 4.

   Convert valid findings into refinement questions. Rephrase each as a neutral question ("Should admins also be notified, or only the payer?") — the model can draft these, but you own the tone. Attach them as Jira comments on each story before the meeting.

5. 5.

   Close the loop in refinement. Every question gets an answer written back into the ACs — an answer that lives only in meeting memory reproduces the original problem. Re-run the critique on rewritten stories if the edits were substantial.

Make it yours

• Claude/ChatGPT project with your Definition of Ready pinnedadd your team's DoR checklist to the system context so the critique scores against your bar, not a generic one.
• Automated draft pass in CI on the backloga scheduled script hits the Jira API for stories entering "Ready for Refinement," runs the critique via an LLM API, and posts findings as a comment tagged ai-draft for your triage. Keep the human-verdict step; never auto-post to the product owner.
• Client-services variantwhere client stories cannot enter a public tool, paste anonymized structure only (replace product nouns with placeholders) or run a local model via Ollama; ambiguity patterns survive anonymization surprisingly well.

Quality check

• Every escalated finding survived your source checkyou re-read the story and any linked spec before raising it; noise findings were discarded silently.
• Cross-story contradictions were verified in both stories, not trusted from the model's memory of the paste.
• Questions are neutral and specifica product owner can answer each in one sentence; none read as criticism.
• Answers landed in the ACscount how many stories were edited after refinement; zero edits means the exercise decorated the meeting instead of changing the sprint.
• False-flag rate is trackedif more than half the model's findings are noise two sprints running, tighten the prompt with examples of your known-context conventions.