Framework and SDK Upgrade Loops with Deterministic Harnesses

Tier: Core · Workflow area: Performance & debugging

This playbook turns a dreaded multi-day upgrade into a supervised loop: codemods handle the mechanical majority, the agent iterates on the ambiguous remainder, and the harness — not the agent's confidence — decides when it is done.

When to use

• •
  A Next.js, React, Expo or React Native major version upgrade with published codemods or an upgrade helper.
• •
  An Android targetSdkVersion bump forced by a Play Console deadline, or an iOS deployment-target raise.
• •
  A large dependency migration (router, styling library, state library) with a documented migration guide.

Skip it for patch or minor bumps that pass CI untouched — merge those from your bot queue and move on.

Prerequisites

• •
  A coding agent that can run shell commands in your repo (Claude Code or equivalent).
• •
  A green baseline: typecheck, test and build all pass on the current version. An upgrade on a red baseline is unfalsifiable.
• •
  The official migration artifacts located up front: Next.js codemods (npx @next/codemod), the React Native Upgrade Helper diff for your exact version pair, npx expo install --fix for Expo, and the Android behavior-changes page for your target API level.
• •
  A dedicated branch and roughly a half day of calendar time with the loop running.

The workflow

1. 1.

   Freeze the harness before touching versions. Write the exact commands the loop must pass and record their green baseline output:

   Copy

   Before we upgrade anything, run these and confirm all pass:
   1. npm run typecheck
   2. npm test -- --ci
   3. npm run build
   Save the list of currently-skipped or known-flaky tests. During the
   upgrade you may NOT skip, delete, or loosen any test or type to get
   to green. If a test seems wrongly failing, stop and report it instead.
   

   That no-loosening rule is the single most important sentence in the prompt.

2. 2.

   Run the mechanical layer first, without the LLM. Codemods and helpers are deterministic; let them do the 80%: npx @next/codemod@latest upgrade for Next.js, the Upgrade Helper diff applied file-by-file for bare React Native, npx expo install --fix plus npx expo-doctor for Expo, and a targetSdk/AGP bump via Android Studio's AGP Upgrade Assistant. Commit this as its own "mechanical" commit so review can separate tool output from model output.

3. 3.

   Hand the agent the residue as a loop. Now start the agentic pass:

   Copy

   The mechanical upgrade is committed. Run the harness. For each failure:
   - Diagnose against the official migration guide (pasted below / linked).
   - Fix the smallest surface that resolves it.
   - Re-run the harness.
   Work until all three commands pass. After each fix, append one line to
   UPGRADE_LOG.md: what failed, why, what you changed. Ask me before any
   change that alters runtime behavior rather than API shape.
   

   Typical residue: renamed APIs the codemod missed, async request APIs in Next.js 15 (await cookies()), removed RN modules that moved to community packages, and Android manifest or permission changes for the new API level.

4. 4.

   Chase behavior, not just compilation, on mobile. A targetSdk bump compiles clean while breaking runtime behavior (background limits, photo picker, exact alarms, foreground-service types). Ask the agent to cross-reference the behavior-changes page against your codebase:

   Copy

   Here is the Android 15 behavior-changes list. Grep the codebase for each
   affected API or pattern and produce a table: change | files affected |
   action needed | risk. Cite file:line for every row.
   

5. 5.

   Verify on device and stage the release. Run the app — web: click the three most valuable flows; mobile: install a release build on a real device, cold start, background/foreground, deep link. Ship behind your normal staged rollout (10% Play rollout, phased App Store release) and watch crash rates for one interval before completing.

Tool variants

• Claude Code / terminal agent: ideal — it runs the harness itself and iterates unattended. Use a max-turns or time budget so a doomed loop stops instead of thrashing.
• Cursor / IDE agent: same loop, but you trigger the harness runs; paste failures back if the agent cannot execute commands.
• Chat-only assistant: paste the failing output batch-by-batch; keep the no-loosening rule in every message, because chat models forget it.

Verification

• •
  All three harness commands pass with zero newly-skipped tests and no weakened types (git diff shows no added @ts-ignore, any, .skip, or deleted assertions).
• •
  UPGRADE_LOG.md explains every non-mechanical change, and you spot-checked three entries against the official migration guide.
• •
  Mobile: release build ran on a physical device through your top three flows; Android: targetSdk behavior table has an owner for every "action needed" row.
• •
  Staged rollout metrics (crash-free rate, ANR rate, Core Web Vitals) match or beat the previous release.

Pitfalls

• Letting the agent "fix" tests to green. The classic failure: it deletes the assertion that was catching the regression. The frozen-harness rule plus a diff review of test files prevents it.
• Mixing mechanical and model commits. One giant commit makes it impossible to tell codemod output (low risk) from LLM judgment calls (high risk).
• Upgrading multiple majors in one loop. Next 13→15 in one pass multiplies ambiguity; go one major at a time, harness-green between each.
• Trusting compilation on Android.targetSdk regressions are runtime-only; step 4 exists because the harness cannot see them.
• Skipping expo-doctor. Expo dependency mismatches surface as bizarre native crashes days later; run it every loop iteration.

Restricted-environment variant

If policy forbids sending source to external APIs: run the codemod/upgrade-helper layer as normal (it is entirely local), then use an enterprise-tenancy assistant (Copilot Business, Bedrock-backed Claude) for the residue loop with the same frozen-harness prompt. With only a local model (Qwen-Coder-class via Ollama), shrink the loop: paste one failing error at a time with the relevant migration-guide section, and keep the harness runs manual.

Success criteria

• •
  The upgrade landed in one branch, in under a day of wall-clock time, with the harness green and no loosened checks.
• •
  Post-release: no crash/ANR/vitals regression attributable to the upgrade after full rollout.
• •
  The UPGRADE_LOG.md became the PR description, and reviewers approved without re-deriving the migration themselves.
• •
  The next upgrade reuses this loop file with only the version numbers changed.