Triage Visual-Regression Diffs with AI Pre-Classification

Tier: Core · Workflow area: Testing & quality

The failure mode of visual testing is not missed pixels — the tools catch everything. It is that catching everything buries the signal. Pre-classification is a triage layer, not a replacement judge: the model sorts, humans decide, and the accept button stops being a reflex.

Prerequisites

• •
  A working visual regression setup with stable baselines: Chromatic, Percy, or Argos (or Playwright toHaveScreenshot in CI), with animations disabled and data seeded in stories.
• •
  Access to a vision-capable model (Claude, Gemini) via API for the classification script.
• •
  The PR context available to the classifier — the intent statement is what makes "intentional" decidable.

The recipe

1. 1.

   Export the diff triples plus the intent. For each changed snapshot, collect baseline image, new image, and the diff overlay if your tool provides one — plus the one thing tools never use: what the PR meant to change:

   Copy

   PR intent: "Bump brand color token from #2563eb to #1d4ed8 and
   increase Button horizontal padding from 12px to 16px. No other
   visual changes intended."
   

   Every visual platform exposes changed snapshots via API or build artifacts; a small script pulls the triples. The intent line comes from the PR description — and making engineers write it is itself a quiet win for review culture.

2. 2.

   Classify with a constrained rubric, one judgment per snapshot.

   Copy

   You are triaging visual regression diffs. PR intent: <above>.
   For this snapshot (baseline, new, diff overlay attached), respond in
   JSON: { "verdict": "INTENTIONAL" | "REGRESSION" | "NOISE" | "AMBIGUOUS",
   "confidence": 0-1, "evidence": "<one sentence naming the specific
   visual change and where>" }
   Rules: INTENTIONAL only if every visible change is explained by the
   stated intent. REGRESSION if any change is outside the intent —
   layout shift, clipped text, missing element, wrong color, overlap —
   even if intent-related changes are also present. NOISE for
   anti-aliasing, sub-pixel shifts, font rasterization differences with
   no layout consequence. AMBIGUOUS whenever you are below 0.8 or the
   intent does not clearly cover the change. Never mark INTENTIONAL by
   assuming an unstated intent.
   

   The asymmetry is deliberate: the model needs positive evidence to clear a snapshot, but any doubt routes to a human. A useful classifier says AMBIGUOUS often; a dangerous one never does.

3. 3.

   Route by verdict — with different bars per verdict. REGRESSION: straight to the top of the human queue with the model's evidence sentence as the caption; these are reviewed first while attention is fresh. AMBIGUOUS: normal human review. INTENTIONAL (high confidence): batched into a collapsed "cleared" section — spot-check a random 10%, then bulk-accept. NOISE: accept, but count — a rising noise rate is your determinism rotting (fonts, animation timing, unseeded randomness) and step 6's input. Humans still press every accept button; the model only decides reading order and grouping.

4. 4.

   Calibrate before you trust. For the first two weeks, run classification in shadow mode: the model labels, humans review everything as before, and you score the disagreements. The one number that matters: model-cleared-as-INTENTIONAL snapshots that a human called regression — that false-clear rate must be at or near zero before you collapse the cleared section, because a false clear is precisely the buried regression this playbook exists to prevent. Model-flagged-but-actually-fine costs minutes; tune the rubric wording and intent quality until the expensive direction is clean.

5. 5.

   Seed-test the pipeline like you would a junior reviewer. Periodically plant a known regression — shift a padding, drop a border, clip a label — inside a large intentional batch and confirm it surfaces as REGRESSION at the top of the queue. A triage layer that has never been adversarially tested is a comfort blanket. This is also the demo that earns the team's trust in collapsing the cleared section.

6. 6.

   Feed the verdicts back into suite health. The classification log is a dataset: stories with chronic NOISE verdicts need determinism fixes (freeze time, mock data, prefers-reduced-motion); components with frequent REGRESSION verdicts are fragile and deserve refactoring or tighter component tests; and intent statements that repeatedly produce AMBIGUOUS verdicts teach engineers to write better PR descriptions. Triage quality compounds when the inputs improve.

Tool variations

• Chromatic: pull changed-story builds via its API; verdicts map cleanly onto its accept/deny per-story model, and the cleared section becomes a bulk-accept.
• Percy / Argos: same pattern via their build APIs; Argos's open API makes the scripted export especially direct.
• Playwright toHaveScreenshot in CI: the DIY lane — failing test artifacts are your triples; the classifier comments its verdict table on the PR, and accepts become --update-snapshots commits.
• Claude Code as the whole loop: for smaller repos, one agent session can pull artifacts, classify, post the verdict table, and open the follow-up determinism fixes — with you as the accept authority throughout.

Quality check

• Intent stated on every visual PRno classification without a written intent; "misc fixes" is not an intent.
• False-clear rate measuredshadow-mode numbers exist, and the INTENTIONAL bar was only trusted after the expensive direction hit zero.
• Humans keep the accept buttonthe model orders and groups; no verdict auto-accepts a snapshot.
• Seeded regressions get caughtthe planted-defect drill passes routinely, not just at rollout.
• Noise trends downthe verdict log drives determinism fixes, and the NOISE rate falls month over month.