Merge HAX-18, PAIR's worksheets, and Shape-of-AI patterns into one working review instrument organized by interaction phase — then run it on a real public AI product, disclosure and dark-pattern checks included. The output is a severity-ranked AX evaluation of portfolio grade: the sellable artifact this track exists to produce.
Keiko has fifteen years of product design behind her and a positioning problem in front of her: every designer now claims "AI experience" on their profile. Her differentiator is the thing almost none of them have — a rigorous, reusable evaluation instrument and a demonstration of it applied to a real product. That is what a heuristic evaluation was for Nielsen-era consultants, rebuilt for the AI era: not vibes about a chatbot, but a phase-structured walkthrough with sourced criteria, evidence standards, and severity discipline. This capstone builds the instrument from the three source frameworks, corrects an AI-merged draft that contains fabrications, and applies the result to a live product. Everything you practiced in Labs 1–14 — citation discipline, planted-error hunting, refusal and oversight patterns, failure taxonomies — converges here.
A previous cohort member asked a model to "merge the frameworks into a checklist." It is the right shape and the wrong content in several places — your first task is the audit.
AX REVIEW INSTRUMENT v0.1 (AI-merged draft, 4 phases)
PHASE 1 — INITIAL (first contact)
1.1 Sets expectations of what it can do (HAX G1)
1.2 Sets expectations of how well it does it (HAX G2)
1.3 Entry points signal capability in context (Shape of AI:
Wayfinders)
1.4 Discloses AI is in use before first output (HAX G20,
"mandatory AI disclosure")
1.5 Onboarding calibrates trust per PAIR's "trust equation"
worksheet (Trust = Competence x Transparency / Stakes)
PHASE 2 — DURING (normal interaction)
2.1 Shows why it did what it did (HAX G11)
2.2 Times services based on context (HAX G4)
2.3 Displays calibrated confidence per Shape of AI's
"Certainty Meter" pattern
2.4 Matches social norms to user context (HAX G5)
2.5 Provides provenance for claims (Shape of AI: Footprints)
PHASE 3 — WHEN WRONG (failure and repair)
3.1 Supports efficient dismissal and correction (HAX G8, G9)
3.2 Scoped services when uncertain (HAX G6 "disambiguate
by default")
3.3 Clear handoff to human/manual path (PAIR: graceful
failure)
3.4 Wrong-but-plausible outputs carry verification
affordances (track principle, Lab 13)
PHASE 4 — OVER TIME (learning and adaptation)
4.1 Learns from behavior transparently (HAX G13)
4.2 Provides global controls over behavior (HAX G17)
4.3 Notifies about changes (HAX G18)
4.4 Memory is inspectable and deletable (PAIR: "right to
reset" principle)
CROSS-CUTTING SWEEP
C1 Disclosure: AI-generated content labeled as such
C2 Dark patterns: no anthropomorphic pressure, guilt, or
fake urgency to keep engaging
C3 Consent: data used for training is opt-in with clear
controls
C4 Severity scale: 0 fine / 1 cosmetic / 2 minor / 3 major
/ 4 blocker
Verify every framework citation in the draft against the primary sources. Three of the draft's attributions are wrong — fabricated numbers, invented pattern names, or misassigned sources. Check each line the way Lab 01 taught you to check quotes: find it in the source or mark it. Log every correction with what the source actually says. Do not skip lines that "sound right"; that is exactly where merged drafts rot.
Rebuild the instrument as v1.0. Keep the four-phase structure — it is the draft's one genuinely good idea. For each criterion: corrected source attribution, a one-line evidence standard ("what would I screenshot to prove a violation?"), and a phase-appropriate probe ("ask the product X and observe Y"). Add anything load-bearing the draft missed — check HAX's mistake-related guidelines and Shape of AI's caution-signaling patterns against your Phase 3, which is thinner than the sources support.
Pre-register your evaluation. Before touching the product: name the product and version/date, list the three tasks you will run (one routine, one at the edge of its competence, one designed to make it fail — you know how to build that from Lab 07), and set your severity definitions in writing. Pre-registration is what separates an evaluation from a browsing session with opinions.
Run Phase 1 and 2 live. First-contact walkthrough: screenshot the empty state, onboarding, entry points; score every Phase 1 criterion with evidence. Then the routine and edge tasks: score Phase 2, capturing provenance behavior, explanation quality, and how the product communicates its degree of certainty in practice.
Run Phase 3 by inducing failure. Execute your designed-to-fail task. Push until you observe at least two of: a wrong or fabricated output, a refusal, an out-of-scope response, a hallucinated citation. Score the repair experience: Could you correct it? Did it acknowledge fallibility? Were you given verification affordances or confident prose? This phase carries the most portfolio weight — reviewers of your report will know instantly whether you actually stress-tested or just chatted.
Run Phase 4 and the cross-cutting sweep. Hunt the memory/personalization surfaces: what does it remember, where is that shown, can you inspect/correct/delete it, did it announce behavior changes? Then the sweep: is AI-generated content labeled where it could be mistaken for human work; any engagement dark patterns (guilt-tripping farewells, fake urgency, sycophancy that discourages leaving); training-data consent findable and honest? Score C1–C3 like every other criterion — evidence or no finding.
Write the severity-ranked report. Structure: product + date + scope; method (instrument version, tasks, pre-registration); findings ranked by severity, each with criterion ID, evidence (screenshot reference + quote), severity with one-line justification, and a concrete recommendation; then a strengths section (an evaluation that finds nothing good reads as a hit piece and marks you junior); close with the three-line executive summary. Use your AI to tighten prose — and audit every product quote in the final text against your screenshots, because a hallucinated quote in an evaluation about hallucination is a portfolio-ending irony.
Package both artifacts. The instrument (v1.0, with your correction log as an appendix — it demonstrates the verification discipline that is this track's brand) and the evaluation report. Together they are the sellable pair: the tool proves method, the report proves application. Write the one-paragraph cover note you would send a prospective client or hiring manager with both attached.
Severity discipline is what graders and clients probe first — a "4 blocker" needs user harm you can articulate in one sentence, and inflating cosmetic findings to look thorough is the fastest way to lose the room.
The capstone artifact is a pair — a phase-structured instrument whose every criterion survives source verification, and an evaluation whose every finding survives evidence audit. Rigor you can hand to a stranger is the product; everything else in this track was training for it.