Run a Skeptical Vendor POC Scorecard

Tier: Advanced · Workflow area: Governance & measurement · ~45 minutes to design; two weeks to run

This playbook designs and runs the two-week structured POC from UC-49. It assumes you can get a trial tenant, an export of your KB, and about an hour a day of desk time — and it produces a scorecard that makes the buy/pass decision boring, which is the goal.

When to use

• Any purchase or renewal above trivial spendif the platform will own part of your queue, it earns two weeks of hostile testing first.
• The vendor quoted a deflection percentage"customers see up to 70%" is a testable claim. The POC is where "up to" meets your ticket mix.
• Two finalists look identical on paperrun the same scorecard on both tenants in parallel; identical marketing rarely survives identical golden sets.

When NOT to use

• When the decision is already madea POC run to ratify a signed deal is theater that costs desk hours. Skip it and instrument hard from day one (pb-44) instead.
• Without your own data in the tenanta POC on vendor sample content tests their demo, not their product. No KB import, no POC.
• For tools with no user-facing claimsa back-office summarizer doesn't need this machinery; a week of side-by-side drafting and pb-44-style spot checks will do.

The recipe

1. 1.

   Freeze the claims sheet first. Before the trial starts, write down every quantified vendor claim from the deck and the sales calls — deflection percentage, setup time, accuracy, languages, "no hallucinations because RAG." One column for the claim, one for how the POC will test it, one left blank for the result. Send it to the vendor; their reaction is itself data.

2. 2.

   Build the golden question set — 60 questions in four buckets. This is the instrument, so build it from pb-42's mined contact types, not from imagination: 25 answerable-easy (top contact types, clean KB coverage); 15 answerable-hard (multi-step, version-dependent, or buried in long articles); 10 unanswerable (real user questions your KB does not cover — the honest answer is "I don't know"); 10 adversarial (ambiguous phrasing, two questions in one, outdated-product mentions, and one prompt-injection attempt pasted from a "user"). For each question, record the verifiable correct behavior — answer plus source article, or the expected refusal.

3. 3.

   Import your real KB and configure like production. Same articles, same permissions model, escalation wired per pb-40's minimums (fallback message, agent keyword). Cap configuration at one working day — a platform that needs three vendor engineers for two weeks to demo well is telling you the operating cost.

4. 4.

   Run the two-week protocol. Week 1: two desk staff run the full golden set through the chat interface, blind-scoring each response — correct-grounded, correct-ungrounded (right answer, wrong/no source), wrong, refused, escalated. Week 2: re-run the full set (consistency check — same question, different day, different answer is a finding), plus 20 live-shadow questions: real tickets from the day, asked verbatim. Log every response with a screenshot; vendors fix trials mid-flight, and you want the before.

5. 5.

   Score the four headline metrics.

   Metric	Computation	Red line
   Strict deflection (projected)	correct-grounded on answerable set × your pb-42 volumes	vendor claim minus 20 pts
   Wrong-answer rate	wrong ÷ all answered (answerable + unanswerable)	> 5%
   Honest-refusal rate	refused ÷ unanswerable 10	< 7 of 10
   Escalation quality	pb-40 rubric: context carried, ≤ 2 turns to human, no dead ends	any dead end

   Wrong-answers on the unanswerable bucket are the number to read aloud in the vendor meeting: every one is a confident invention delivered to a user.

6. 6.

   Compute cost per resolution, fully loaded. (License + per-conversation/token fees + integration effort amortized + your admin hours at loaded cost) ÷ projected strictly-deflected tickets per year. Compare against your current cost per L1 ticket. A platform that deflects 25% at ₹700 per deflected ticket when your L1 cost is ₹450 is a loss dressed as innovation.

7. 7.

   Write the one-page verdict against the claims sheet. Each frozen claim gets its measured result and a verdict: held / partially held / failed. Then the recommendation — buy with conditions (name them: e.g., "refusal behavior must be configurable before go-live"), extend the POC (name the single question to resolve), or pass (attach the numbers). File the whole pack; it is your negotiation leverage now and your pb-44 baseline later.

Make it yours

• Comparing two finalistssame golden set, same weeks, same scorers, tenants run in parallel; add a fifth metric, admin effort hours, tracked per platform. The gap there often decides more than accuracy does.
• Renewal, not new purchasethe "trial tenant" is production: run the golden set against the live bot and score the incumbent with the same hostility. Incumbents fail golden sets more often than procurement expects.
• No trial tenant offeredthat is a scorecard result in itself. Push for a paid pilot with an exit clause keyed to the red lines above; a vendor who resists measurable pilot criteria has told you the POC result in advance.

Quality check

• The golden set stays sealed. Never share the questions with the vendor before or during the trial; a set they have seen measures their tuning speed, not their product.
• Scoring is blind and doubled. Two scorers, rubric agreed in advance, disagreements adjudicated in writing — same discipline as pb-44's review loop.
• Both weeks' runs are kept. The week-1 to week-2 delta on identical questions is your consistency metric and your evidence if trial behavior mysteriously improved.
• The desk staff who scored get a veto voice. They saw the tool at its worst; procurement saw the demo. The verdict page carries both signatures.