Evaluate AI Vendors on the Questions SaaS Procurement Never Asks

Workflow area: Measurement & ROI · Tier: Core · Hands-on time: ~35 minutes

The failure mode is signing a standard SaaS agreement for a non-standard product. Six months later you discover the default tier trains on your prompts, the model your workflows depend on is deprecated with 90 days' notice, your data sits in a region your customers' contracts forbid, and the indemnity clause explicitly excludes AI-generated output. None of this was hidden — it was in the DPA and the trust center, unread, because procurement's checklist predates the category. Engineering has to bring the AI-specific questions, because nobody else in the room knows to ask them.

When to use — and when not to

• Use it for any AI tool touching source code, internal documents, or customer datacoding assistants, chat tools, meeting AI, agent platforms.
• Use it at renewal, not just purchaseAI vendors change training defaults, model lineups and terms far faster than classic SaaS; last year's answers may be stale.
• Use it when a team requests an unsanctioned toolthe scorecard turns "no" into "here's what it would need to pass," which is a conversation instead of a wall.
• Skip the full weighted treatment for free-tier experiments on public dataa two-line check (does it train on inputs? where does data go?) gates a sandbox trial; save the scorecard for anything touching real work.
• Don't outsource the whole evaluation to procurement or security alonethey will correctly assess the vendor and miss whether the product fits your workflows; you own the joint verdict.

The recipe

1. 1.

   Build the weighted scorecard once, reuse it for every vendor. Six categories, weights agreed with security and procurement up front:

   Copy

   AI VENDOR SCORECARD (score each 1-5, multiply by weight)
   25% DATA USE & TRAINING — Is training on customer data off by
       default at our tier? Contractually, not just in settings?
       Zero-data-retention option? Who owns fine-tunes on our data?
   20% RETENTION & RESIDENCY — Prompt/output retention window and
       deletion SLA. Region pinning available? Subprocessor list —
       which model providers actually see our data?
   15% SECURITY & COMPLIANCE — SOC 2 Type II / ISO 27001, SSO+SCIM,
       audit logs, tenant isolation. (The classic checks still count.)
   15% INDEMNITY & LIABILITY — IP indemnity that covers AI OUTPUT,
       not just the service. Caps. Who is liable if generated code
       infringes?
   15% CONTINUITY & LOCK-IN — Model deprecation policy and notice
       period. Export of prompts, configs, and history in usable
       form. Can workflows survive a model swap?
   10% ADMIN & COST CONTROLS — usage reporting, per-team budgets,
       seat management (feeds pb-40).
   Kill questions (any "no" ends the evaluation): training opt-out
   at our tier; breach notification duty; data deletion on exit.
   

2. 2.

   Make the vendor's documents answer it. Collect the trust center pages, DPA, security whitepaper, and a filled SIG/CAIQ questionnaire if they have one, then extract:

   Copy

   You are evaluating an AI vendor against the attached scorecard.
   Here are their DPA, trust-center pages, and security docs.
   
   For each scorecard question:
   - ANSWER with a direct quote from the documents and a citation
     (document + section).
   - If the documents are silent or ambiguous, write [UNANSWERED]
     — do not infer generosity the text doesn't state.
   - Flag contradictions between marketing pages and the DPA with
     [CONFLICT] and quote both.
   
   Then list the [UNANSWERED] and [CONFLICT] items as questions to
   send the vendor, phrased for a written response.
   
   Documents: [PASTE / ATTACH]
   

   The [UNANSWERED] list is the real output. Good output looks like:

   Copy

   DATA USE — Training off by default? [CONFLICT]
   - Marketing FAQ: "We never train on your data."
   - DPA §4.2: "...may use Customer Content to improve Services
     unless Customer opts out via account settings."
   → Question for vendor: confirm training opt-OUT is contractual
     default for our tier, in writing, in the order form.
   

3. 3.

   Get the gaps answered in writing, in the contract. Settings-page promises change with a product update; order-form language survives. Anything that scored on the basis of a sales call gets re-scored to what the contract actually says.

4. 4.

   Run a two-week structured pilot before signing. Scorecards assess the vendor; only usage assesses the product. Give 5–10 engineers real tasks, collect the same feedback form daily, and check admin claims empirically — do the audit logs actually show what the docs promise?

5. 5.

   Score, decide, and file the record. Compute the weighted score, write a one-page decision memo (score table, kill-question results, pilot verdict, dissents), and calendar a renewal re-evaluation. The memo is what saves you when someone asks "why did we pick this?" — or when the vendor's terms change and you need to show what you signed against.

Make it yours

• Startup or small team: you won't get contract redlines from big vendors — so weight RETENTION and DATA USE even harder and choose tiers where the defaults are already right (business/enterprise tiers with training off) rather than negotiating. The scorecard still runs; it just selects between offerings instead of amending them.
• Mid-size org: run the scorecard jointly — engineering scores product fit and continuity, security scores data and compliance, procurement scores commercial terms — and require the memo before any AI tool gets an SSO integration.
• Large org or GCC: fold the scorecard into your formal vendor-risk process as an AI addendum to the SIG/CAIQ, maintain a pre-approved vendor list with tier constraints, and re-run kill questions at every renewal — parent-company data-residency commitments to your clients make the residency and subprocessor questions contractual, not preferential.

Quality check

• Every score cites a document, not a demoa scorecard filled from sales-call memory is the demo with extra steps.
• Kill questions were actually lethalif a vendor failed one and got waived through anyway, the scorecard is theater and everyone learns to treat it that way.
• The DPA beat the marketing page wherever they disagreed[CONFLICT] items resolved toward the contract, in writing.
• The pilot tested your workflows, not the vendor's demo scriptreal repos, real tickets, your security constraints on.
• A renewal date is on the calendar with a re-check ownerAI vendor terms drift; an evaluation without an expiry date silently becomes false.