Run a One-Day AI Feasibility Spike Before Committing Roadmap

Workflow area: AI Product Economics · Tier: Advanced · Hands-on time: ~45 minutes to learn, one day to run

The most expensive sentence in AI product development is "the model should be able to do that." Teams commit a quarter to a feature on the strength of one impressive ChatGPT exchange, discover in week eight that quality plateaus at 70% on real data, and ship a disappointment or quietly cancel. Almost everything that kills AI features — messy real-world inputs, knowledge gaps, hallucination under ambiguity, latency — is discoverable in one day with zero engineering, if you spike it honestly. The spike is a PM-runnable experiment: prompt, twenty real cases, graded outputs, verdict.

When to use — and when not to

• Use it before any AI feature enters a committed roadmapthe spike is the toll bridge between "great idea" and "resourced quarter."
• Use it to arbitrate between competing AI feature ideasthree spikes in three days produce a ranked list backed by graded evidence instead of stakeholder enthusiasm.
• Use it when a stakeholder brings the impressive demo"let's spike it on our data this week" is the constructive version of skepticism, and it either upgrades the idea or retires it gracefully.
• Skip it for features already proven at your scale by close competitors on similar dataspike the differentiated part, not the commodity part.
• Skip the verdict if you only ran happy-path casesa spike on clean inputs is the demo problem recreated at home; the messy-case quota below is non-negotiable.

The recipe

1. 1.

   Write the kill criteria first. Before touching a model, one paragraph: what does the feature need to do, on what inputs, and what result would kill the bet? Example: "Summarize inbound enterprise support threads into case briefs. Kill if: under 70% of briefs are usable-with-light-edits on real threads, or critical facts (account tier, deadline) are dropped/invented in more than 1 of 20." Deciding the bar after seeing outputs is how mediocre results get rationalized into roadmaps.

2. 2.

   Assemble 20 real cases, with a messiness quota. Pull from production: 12 typical, 5 hard-but-plausible (long threads, mixed languages, angry customers, missing context), 3 known nightmares. Scrub PII first. Twenty real cases beat two hundred synthetic ones — the entire point is contact with your actual data distribution.

3. 3.

   Build the interaction in a workbench, iterating the prompt honestly. In the Anthropic Workbench (or your provider's playground), write the system prompt as if it were production: role, task, format, refusal instructions. Run cases, improve the prompt, but log every iteration — if case 7 only passes after you tuned the prompt specifically for case 7, that's overfitting and the log will show it.

   Copy

   SPIKE LOG — thread-summarizer, prompt v3
   Case 04 (typical): PASS — usable with no edits
   Case 09 (typical): PASS — light edit (tone)
   Case 13 (hard: 40-message thread): FAIL — dropped the renewal
     deadline; deadline present at msg 22
   Case 17 (hard: mixed EN/HI): PARTIAL — summary fine, quoted
     customer line mistranslated
   Case 19 (nightmare: forwarded email chain): FAIL — invented an
     SLA commitment nobody made  ← S1-type failure
   

4. 4.

   Grade against the kill criteria, not against hope. Score all 20 with per-case verdicts and one line of reasoning. Tally: pass rate overall, pass rate on hard cases, and — separately — any invented-fact failures, because a 10% hallucination rate on commitments is a different problem than a 10% clumsiness rate. If you can, run the set through promptfoo so the spike's cases become the seed of the real eval set (pb-37) — the spike's best byproduct.

5. 5.

   Check the two non-quality gates. Latency: time five representative runs; if the interaction takes 20 seconds and the UX needs 3, that's a design constraint to surface now. Cost: tokens per case × projected monthly volume × price — pb-46's math at napkin fidelity. A feature that works at $0.40/interaction against a $0.05 budget has a routing/model-tier problem to solve before commitment, not after.

6. 6.

   Write the one-page verdict memo. Recommendation (commit / commit-with-conditions / kill / needs-a-second-spike), the kill criteria as written before the spike, the score table, the failure examples verbatim (nothing communicates like case 19's invented SLA), latency and cost napkin math, and what a committed team would need to solve first. A kill verdict is a success — say so in the memo's framing: the spike just bought the roadmap a quarter.

Make it yours

• The RAG variant: if the feature needs your knowledge base, spike retrieval too — paste the relevant docs into context for each case (manual RAG) and grade whether answers ground correctly; retrieval quality, not generation, is where most knowledge features die.
• The UX-attached spike: wire the working prompt into a Bolt or v0 front end in an afternoon and put it in front of three users — only when interaction design is the risky part; don't let prototype polish launder weak quality numbers.
• The spike week: institutionalize it — every AI roadmap candidate gets a spike slot before quarterly planning, and planning reviews verdict memos instead of pitch slides.

Quality check

• Kill criteria were written before any model was rundated, in the memo, unedited.
• Cases are real and the messiness quota was metproduction-drawn, PII-scrubbed, with hard and nightmare cases included and labeled.
• Prompt iterations were loggedper-case tuning is visible, and the final score comes from one frozen prompt version run across all cases.
• Invented facts are counted separately from clumsinesshallucination failures are named, quoted and severity-tagged in the memo.
• Latency and cost got napkin math at projected volume, against explicit budgets.
• The memo renders a verdict against the pre-written barincluding "kill," framed as the spike doing its job.