Lab 03 · Registry-Grounded Generation A/B

Priya's team standardized on shadcn/ui a year ago. Last sprint a teammate merged an AI-generated admin table that imported @/components/ui/data-table — a module that has never existed in their repo — with a sortable prop the real Table has never had. It compiled on his machine because the agent had also helpfully invented the missing file. The PR looked plausible, reviewed plausible, and broke design consistency in three places. This lab makes that failure mode measurable: you run the same generation task grounded and ungrounded, and you count the hallucinations yourself.

The mechanism behind the hallucination is worth naming before you reproduce it. An ungrounded model answers "build a shadcn data table" from a statistical composite of every table library it has ever seen — Material UI, Ant Design, TanStack, react-table, a thousand blog tutorials — blended into a plausible average that resembles no specific library's real API. It is not lying; it is interpolating, and interpolation across libraries is exactly where invented props and lookalike modules come from. A registry MCP breaks the interpolation by giving the agent a tool to look up the ground truth — the actual installed component's props, the actual variant names, the actual install command — so it writes against one real API instead of a memory of many. The A/B you are about to run is the cleanest possible demonstration of that difference, because the only variable that changes between the two runs is whether the agent could consult reality.

Time & tier

~60 minutes hands-on · Build stage (maps to module M2, playbook PB-03) · $0 — Claude Code, Cursor, or any MCP-capable agent; shadcn registry MCP is free.

Setup

npm create vite@latest registry-lab -- --template react-ts
cd registry-lab && npm install
npx shadcn@latest init   # accept defaults (slate, CSS variables)
mkdir -p runA runB

Wire the shadcn registry MCP for run B only. For Claude Code, .mcp.json in the repo root:

{
  "mcpServers": {
    "shadcn": {
      "command": "npx",
      "args": ["shadcn@latest", "mcp"]
    }
  }
}

(Cursor: add the same server under Settings → MCP. Verify it's live by asking the agent "list available registry items" — you should see table, badge, dropdown-menu, etc.)

The task prompt — identical for both runs, save it as task.md:

Build src/OrdersTable.tsx: an orders admin table using this project's
shadcn/ui components.

Data shape:
  { id: string; customer: string; total: number;
    status: "paid" | "pending" | "refunded"; placedAt: string }

Requirements:
- columns: Order ID, Customer, Total (right-aligned, currency),
  Status (colored badge per status), Placed (relative date)
- click a column header to sort by that column
- a row-actions menu (View, Refund) on each row
- empty state when there are no orders
- seed 8 rows of demo data inline

And the memo template, comparison-memo.md:

# Registry grounding A/B — comparison memo

## Run A (ungrounded) — claims audit
| # | Claim in generated code | Real? (checked against actual API) | Class |
|---|--------------------------|-------------------------------------|-------|
Classes: HALLUCINATED-PROP / HALLUCINATED-MODULE / INVENTED-LOOKALIKE /
         WRONG-PATTERN / OK

## Run B (registry-grounded) — install audit
Components actually installed (from registry):
Props verified against installed source:

## Verdict
Compile result A vs B:
Visual/design-system consistency A vs B:
Time to working screen A vs B:
One-paragraph recommendation for the team:

Guided steps

1. 1Run A — ungrounded. Disable or omit the MCP server (move .mcp.json aside). Fresh agent session, paste task.md, and let it generate without asking clarifying questions. Save the output to runA/OrdersTable.tsx plus any extra files it invents. Do not correct it, do not install anything it tells you to install unless the package genuinely exists and the agent's code needs it to compile — record every such intervention. Expected: plausible-looking code in 2–4 minutes.
2. 2Audit run A against reality. Try to compile: npx tsc --noEmit. Then check every import path and every prop against the actual installed source in src/components/ui/ (for components it references but never installed, check the real shadcn docs or registry source). Fill the Run A claims table — one row per import and per non-HTML prop. Expected: 10–20 rows. Be strict: a prop that exists on some other library's Table but not on this project's is HALLUCINATED-PROP, not OK.
3. 3Run B — registry-grounded. Restore .mcp.json, start a fresh session, paste the identical task.md with one added line: "Use the shadcn registry MCP to look up and install any components you need before writing code." Expected behavior worth watching: the agent queries the registry, runs real installs (npx shadcn@latest add table badge dropdown-menu), reads the installed source, then writes against the real API — sorting implemented in userland state or via @tanstack/react-table, because it discovered the primitive Table has no sortable prop.
4. 4Audit run B the same way.npx tsc --noEmit, then verify a sample of props in runB/OrdersTable.tsx against the installed component source. Fill the install audit. Expected: imports resolve to files that exist because the agent installed them; prop usage matches the installed signatures.
5. 5Diff the two outputs side by side.diff runA/OrdersTable.tsx runB/OrdersTable.tsx and a visual skim. In the memo, note the structural differences: where run A invented an API surface, what run B did instead with the real primitives.
6. 6Make run B production-real. Wire OrdersTable into App.tsx, npm run dev, and manually exercise: sort toggles on at least two columns, the actions menu opens and is keyboard-reachable, the empty state renders when you pass []. Fix any residual issues with the agent, keeping it grounded ("check the installed dropdown-menu source before answering").
7. 7Write the verdict. Complete the memo's verdict section, including the one-paragraph team recommendation: when is ungrounded generation acceptable (throwaway exploration), and what does your team gate on for production code (registry grounding + compile + prop audit)?

Verify

• •
  Both raw outputs preserved unmodified in runA/ and runB/; every intervention you made is logged.
• •
  Run A claims table has 10+ rows, each verdict backed by a check against real installed source or registry docs — not memory.
• •
  Run B's screen compiles (tsc --noEmit clean) and passes the three manual checks: sorting, keyboard-reachable row actions, empty state.
• •
  The memo's verdict section is complete, including compile results for both runs and the team recommendation paragraph.
• •
  At least one INVENTED-LOOKALIKE or HALLUCINATED-MODULE finding is documented (if run A was miraculously clean, rerun it with a different model and say so in the memo).

Self-grading rubric (pass = 7/10)

• Audit rigor (0–4): every run-A import and prop row verified against actual source, correct class assigned = 4; spot-checked only = 2.
• Working screen (0–3): run B compiles and all three manual checks pass = 3; compiles but a check fails = 1.
• Memo quality (0–3): verdict quantifies the difference (counts, compile results) and the recommendation draws a defensible grounded/ungrounded boundary = 3.