Evaluate Vendor AI-Testing Claims with a Two-Week POC Scorecard

Tier: Core · Workflow area: Quality leadership & governance

This playbook structures a proof-of-concept for mabl, testRigor, Applitools or any AI-testing platform: pre-registered metrics, a controlled comparison against your existing suite, deliberately seeded UI changes and bugs, and a scorecard the vendor sees on day one — because a POC without pre-agreed numbers becomes a demo with your data.

When to use

Prerequisites

• •
  A trial tenant with enough seats and runs for real load, in writing from the vendor.
• •
  Your control: an existing suite (10–30 stable tests) on a staging build you can modify — seeded changes require deploy control.
• •
  Two evaluators who will do the actual work (not the most enthusiastic tool fan alone), with calendared time.
• •
  The scorecard drafted and shared with the vendor before day one.

The workflow

1. 1.

   Translate each marketing claim into a measurable question. Three or four claims maximum:

   Copy

   Claim: "Cuts test maintenance by 90%"
   Question: Across 12 seeded UI changes, how many tests does each
     stack repair without human edits, and what is the human minutes
     per repair for the rest?
   
   Claim: "Self-healing keeps suites green"
   Question: Of the heals performed, how many were CORRECT vs
     WRONG (test passes while seeded bug is live)?
   
   Claim: "Anyone can author tests in plain English"
   Question: Minutes for a manual tester to author 5 specified
     scenarios, and how many assert real behavior vs presence-of-text?
   

   The wrong-heal question is non-negotiable for any self-healing claim — it is the failure mode vendors do not demo (see pb-17).

2. 2.

   Pre-register the scorecard. Metrics, weights, and the decision rule, agreed with your own stakeholders and shared with the vendor before the trial:

   Copy

   POC scorecard — agreed 2026-08-10, before trial start
   Metric                          Weight  Source
   Authoring min/test (5 scenarios)  15%   timed sessions, both evaluators
   Maintenance min across seed set   25%   12 seeded changes
   Wrong-heal count                  20%   4 seeded bugs paired w/ changes
   Real-coverage score               15%   planted-bug detection (8 bugs)
   Flake rate over 50 CI runs        10%   scheduled runs, same suite
   Exportability / lock-in           10%   export tests, run outside? Y/N
   Support responsiveness             5%   trial tickets, hours to useful
   Decision rule: adopt if weighted score beats control by >=20%
   AND wrong-heal count is 0-1 with all heals surfaced for review.
   

   Pre-registration is the integrity mechanism: metrics chosen after results are advocacy, not evaluation.

3. 3.

   Build the challenge set into staging. Three seeded packages, prepared before the trial and revealed to nobody who evaluates:

   • •
     12 UI changes across the spectrum self-healing must handle: renamed test-ids, restructured DOM, moved elements, changed button text, a redesigned form.
   • •
     4 real bugs paired with changes — the wrong-heal traps: e.g., the submit button renamed and its handler broken. A healer that repairs the locator and goes green has actively hidden a defect; count it.
   • •
     8 planted functional bugs for the coverage question — wrong totals, broken validation, missing error states — measuring what each stack's tests actually catch, not what they execute.
4. 4.

   Run week one: authoring and baseline. Both evaluators author the same 5 scenarios in the vendor tool (timed), replicate your 10–30 control tests, and start the 50-run flake series in CI. Log every support ticket. Resist vendor "let our SE build it for you" offers for the timed portions — you are measuring your team's experience, not the sales engineer's.

5. 5.

   Run week two: the challenge set. Deploy the seeded changes in batches, let both stacks (vendor suite and your control suite) run, and record per change: auto-repaired / human-fixed (minutes) / broken. Then check every heal against the paired-bug list:

   Copy

   Seed batch 3 results
   Change C-07 (rename #submit -> #order-submit + broken handler)
     Control suite: FAILED (correct — bug is live)
     Vendor suite:  PASSED after heal    <- WRONG HEAL, logged
   Change C-08 (moved nav into drawer)
     Control: 20 min human fix   Vendor: auto-healed, verified correct
   

6. 6.

   Score, decide, and write the memo. Fill the pre-registered scorecard, apply the decision rule as written, and record the result in a two-page memo: numbers, the wrong-heal narrative, exportability findings, and the recommendation. If the vendor loses, the memo is your renewal-negotiation leverage; if it wins, the memo's seeded-change harness becomes your regression check at every major platform update — vendors change their AI models too, and this year's heal accuracy is not a contract for next year's.

Variations

• Head-to-head (two vendors): same challenge set, same scorecard, same weeks — commensurability is the entire value; resist letting each vendor propose their own success criteria.
• Incumbent renewal: run the challenge set against the incumbent alone; the wrong-heal and real-coverage numbers tell you what you are actually renewing.
• Visual-AI tools (Applitools-class): swap the seeded-change set for a visual one — intended redesigns vs seeded rendering bugs — and score ML-diff triage accuracy and baseline-management overhead instead of locator healing.

Quality check

• Scorecard dated before the trial startedthe git timestamp is your receipt.
• Wrong-heal traps present and countedany self-healing evaluation without them is theater.
• Control suite ran the same challengesvendor numbers without a baseline are unanchored.
• Timed work done by your evaluators, including at least one skeptic, not by the vendor's SE.
• Exportability tested by doing itactually export and run the tests elsewhere, not by reading the docs page.

Pitfalls

• The perpetual demo. Vendor-driven POCs optimize for wow moments on their sample app. Your app, your data, your evaluators, your seeded chaos.
• Measuring week-two enthusiasm. Novelty masks maintenance cost — that is exactly why the seeded-change protocol exists; without it a two-week trial structurally cannot see the claim that matters most.
• Ignoring lock-in until renewal. The exportability score at 10% looks small until year two's pricing arrives. Test the exit on the way in.
• Letting a good score waive governance. An adopted tool's heals still go through the audit queue (pb-18); the POC measures the tool, it does not retire your gates.