AI drafts a technology comparison in minutes — and salts it with invented service limits, stale SLAs, and regions where the service does not exist. This playbook makes verification a formal step: every quantitative claim goes into a claims register and is checked against vendor documentation before the brief leaves your laptop.
Tier: Core · Hands-on time: ~30 minutes
Hallucinated service capabilities are this discipline's signature AI failure mode. An architect who ships an AI-invented quota into a design is worse off than one who never used AI, because the error arrives wearing the confidence of a finished document. LLM recall of limits and pricing is unreliable by construction — training cutoffs guarantee staleness even when the model was once right — so the fix is not a better model but a better process: draft fast, then verify every number against the primary source, today.
Skip AI when the decision is already made and documented — write the ADR (pb-14) instead of retro-fitting a comparison to justify it.
Frame the evaluation before drafting. Write down the candidates, the deciding criteria, and the disqualifiers. Anita, evaluating vector stores for Orbit's docs copilot, frames it as: pgvector on the existing Postgres vs a managed search service vs a dedicated vector database — criteria: co-location with operational data, filter support, scale ceiling, cost floor, EU-region availability.
Draft the brief with claims pre-tagged. Make the model do the bookkeeping that verification needs:
Draft a technology-evaluation brief comparing pgvector (on our existing
managed Postgres), a managed cloud search service with vector support,
and a dedicated managed vector database, for a B2B SaaS documentation
copilot ("orbit-docs-copilot"): ~2M chunks at launch, metadata
filtering per customer tenant, EU data residency required.
Rules:
- Tag EVERY quantitative or capability claim inline as [CLAIM-n]:
limits, quotas, SLAs, pricing, regional availability, feature
support, version requirements.
- After the brief, emit a claims register table: CLAIM-n, the exact
claim, which vendor doc page would confirm it (name the page, do
not fabricate a URL), and your confidence (high/medium/low).
- Where you are unsure, write "UNVERIFIED" in the claim text itself
rather than a plausible number.
- Structure: one page per option (fit, strengths, risks, exit cost),
then a comparison table against my criteria.
###
[paste context]
Verify every claim against the primary source — today. Open the real vendor pages and work the register row by row: confirm, correct, or strike. Record the actual URL and today's date. Rules of the step: a claim with no findable source is struck, not softened; a corrected claim gets the doc's number, not a compromise between the doc and the model; low-confidence tags from the model are checked first but high-confidence tags are not exempt — models are confidently wrong more often than they are hesitantly wrong.
Check the claims the model did not make. Verification has a second half: what is missing? Walk your criteria list against each vendor's limits page directly — per-tenant index ceilings, request-rate quotas, filter-cardinality limits. The costliest evaluation failures are silent: the constraint nobody mentioned until load testing. Add what you find as new, human-authored register rows.
Rewrite the brief from the verified register. Strip the [CLAIM-n] tags, fold in corrections, and add a "Verification" footer: "All limits, pricing and availability claims verified against vendor documentation on [date]; register attached." That footer converts the brief from AI-drafted prose into an engineering document.
Set an expiry. Cloud limits and pricing change without notice. Date the brief, and state that decisions taken more than a quarter later require re-verification of the register. A stale verified brief is only marginally better than an unverified one.
Evaluation briefs often carry client context (scale figures, regions, compliance posture) that the pb-03 register may restrict. Redaction works well here: the comparison logic needs workload shape, not the client's name — "US retailer, ~40K support tickets/month" carries no identity. For fully restricted engagements, draft with a client-tenancy endpoint (Bedrock/Azure OpenAI in the client subscription) or a local Ollama model, and note that the verification step needs no AI at all — vendor docs are public, and the claims register is a browser-and-spreadsheet exercise you can run anywhere.
A technology brief is only as trustworthy as its oldest unverified claim. If the claims register does not carry today's date, the brief is a rumor with formatting.