Run a One-Day Feasibility Spike Before Anyone Writes a Business Case

Workflow area: AI Opportunity & Feasibility · Tier: Advanced · Hands-on time: ~40 minutes to set up; a day to run

Roughly 95% of GenAI pilots show no P&L impact, and a large share of those were doomed at conception: the model was never going to hit usable quality on that task with that data. The feasibility spike is the BA's cheapest instrument for finding that out first. It is deliberately crude — a prompt in a workbench, real cases pasted in, outputs scored by hand — because crudeness is the point: if the interaction cannot look promising under generous conditions, it will not survive production conditions.

When to use — and when not to

• Use it on every shortlisted opportunity before the business casethe spike verdict ("plausibly reachable" or not) is an input the business case should cite, not a step that follows it.
• Use it to convert a skeptical or over-excited sponsorthirty real cases with scored outputs beats both "AI can't do this" and "AI will solve this" in one meeting.
• Use it to discover the real requirementsthe cases the spike fails on become your failure-mode catalog, eval set seeds, and oversight requirements later.
• Skip it when the task needs data you cannot legally sampleif you cannot get 30 anonymized real cases, you have a data-readiness problem to solve first (pb-39), not a feasibility question.
• Skip it for use cases where feasibility is not the questionretrieval over your own documents with citations is known-feasible; spike effort belongs on the genuinely uncertain interaction.
• Do not let the spike masquerade as the pilota day of prompting proves plausibility, not production quality; say so in writing every time you present it.

The recipe

1. 1.

   Sample 20–30 real cases with an SME. Pull them from the actual queue the AI would work — not the examples the sponsor remembers. Force the mix: mostly routine, several edge cases, at least three where the correct behavior is "don't answer, escalate." Anonymize. For each case, have the SME write the expected outcome in one line before any model runs — this is your answer key, and writing it first keeps the scoring honest.

2. 2.

   Write the task prompt like a requirement, not a wish. Draft it in a workbench (Claude or OpenAI console — no code needed). A serviceable spike prompt skeleton:

   Copy

   You are assisting with [TASK — e.g. "triaging inbound insurance
   claims emails"] for [CONTEXT — one line on org and process].
   
   For the input below, produce:
   1. CLASSIFICATION: one of [your real category list]
   2. EXTRACTED FIELDS: [the fields the process needs], each with
      the exact source phrase it came from
   3. CONFIDENCE: high / medium / low, with the reason
   4. If required information is missing or the case is ambiguous,
      output ESCALATE and say what is missing. Never guess.
   
   Rules: use only the input text. Do not invent policy numbers,
   dates, or names. Quote, don't paraphrase, in extracted fields.
   
   Input:
   [PASTE CASE]
   

3. 3.

   Run all cases and score against the answer key. One row per case in a spreadsheet: case ID, expected, actual, score (pass / partial / fail), and a one-line failure reason. Iterate the prompt when you see systematic misses — that is legitimate; you are testing reachability, not first-attempt luck — but rerun the full set after each prompt change, and keep the version history. Stop after a day.

4. 4.

   Read the failures as an analyst. Tally the spike scoreboard:

   Copy

   SPIKE RESULT — claims email triage, 28 cases, prompt v4
   Pass 21 · Partial 4 · Fail 3      (75% clean pass)
   Escalation behavior: 3/3 ambiguous cases correctly ESCALATEd
   Failure pattern: all 3 fails are multi-claim emails — model
     merges two claims into one record
   Verdict: PLAUSIBLY REACHABLE with a multi-claim split step;
     multi-claim handling enters requirements as R-candidate.
   

   The pattern matters more than the percentage — one fix-able systematic failure is encouraging; scattered random failures on routine cases are the death signal.

5. 5.

   Write the half-page verdict memo. Verdict (proceed to business case / proceed with named prerequisites / kill), the scoreboard, the failure patterns, the prompt version that produced the result, and the explicit caveat: generous conditions, hand-fed inputs, no integration, no adversarial users. Attach the case set — it seeds the golden dataset (pb-48) if the initiative proceeds.

Make it yours

• Workbench-only for zero setup: paste cases one at a time into the provider console; slow but universally available and MSA-friendly if you use the org's approved workspace.
• promptfoo when you want the reruns cheap: cases as a CSV of test inputs, expected values as assertions — every prompt iteration re-scores all 30 cases automatically.
• Spreadsheet + LLM for the scoring grind: paste actual-vs-expected pairs and let a model pre-score them against your rubric, then hand-verify every fail and a sample of passes — never outsource the final score unchecked.
• At program scale: make the spike a stage gate — no AI initiative enters the portfolio without a spike memo, and collect the verdicts; the kill list teaches the org more than the wins.

Quality check

• Cases are sampled, not curatedpulled from the real queue with the ugly ones in; a spike on cherry-picked inputs is a demo you ran on yourself.
• Answer key was written before outputs existedexpected outcomes are the SME's, uncontaminated by what the model produced.
• Escalation cases included and scoreda model that answers confidently when it should escalate has failed feasibility even at 90% accuracy elsewhere.
• Full set rerun after every prompt changeno verdict rests on a version that only saw half the cases.
• Failure patterns named, not averagedthe memo says what kind of case fails, because that sentence becomes a requirement.
• The caveat is in writingthe memo states this proves plausibility under generous conditions, and someone senior has read that sentence.