Lab 07 · Legacy Upgrade Combo: Characterize, Then Migrate

This is the two-phase pattern behind Google's and Uber's LLM-assisted migrations: deterministic verification first, mechanical transformation second. Skipping phase 1 is how AI migrations silently change behaviour.

Time & tier

~2 hours hands-on · Advanced.

Setup

Inline scaffold — a real breaking-change upgrade, no generation prompt needed. Create pricing/ and install the old pin: pip install "pydantic==1.10.26" pytest pytest-cov mutmut (1.10.26 is the last v1 release and installs cleanly on modern Pythons).

pricing/__init__.py: empty. pricing/models.py:

from pydantic import BaseModel, validator
from typing import Optional, List

class LineItem(BaseModel):
    sku: str
    qty: int = 1
    unit_price: float

    @validator("qty")
    def qty_positive(cls, v):
        if v < 1:
            raise ValueError("qty must be >= 1")
        return v

    @validator("sku")
    def sku_norm(cls, v):
        return v.strip().upper()

class Quote(BaseModel):
    customer_tier: str = "standard"
    items: List[LineItem]
    coupon: Optional[str]

    class Config:
        anystr_strip_whitespace = True
        allow_population_by_field_name = True

def quote_total(q: Quote) -> dict:
    subtotal = sum(i.qty * i.unit_price for i in q.items)
    disc = 0.10 if q.customer_tier == "gold" else 0.0
    if q.coupon == "WELCOME5" and subtotal >= 50:
        disc += 0.05
    total = round(subtotal * (1 - disc), 2)
    return {"subtotal": round(subtotal, 2), "discount": disc,
            "total": total, "items": [i.dict() for i in q.items]}

def parse_quote(raw: dict) -> Quote:
    return Quote.parse_obj(raw)

Verification checklist: python -c "from pricing.models import parse_quote, quote_total; print(quote_total(parse_quote({'items':[{'sku':' ab1 ','qty':2,'unit_price':30}], 'coupon':'WELCOME5'})))" prints a total of 57.0 with sku AB1; pip show pydantic says 1.10.26; there are zero tests. The module uses at least four v1-only idioms (@validator, class Config, .dict(), parse_obj) — confirm by grep; that's the migration surface.

Guided steps

1. 1Phase 1 — characterize. Prompt your assistant (Copilot Free chat, Gemini Code Assist, Windsurf) with the module and this contract: "Write pytest characterization tests for this module. Rules: assert what the code DOES today, not what it should do. Cover: sku normalization, qty validation error, whitespace stripping from Config, coupon boundary at subtotal exactly 50, gold-tier stacking, a payload that omits the coupon key entirely, a customer_tier with padded whitespace, the exact dict shape returned, and parse_quote with unknown extra keys. No fixes, no opinions." Save to tests/test_characterization.py.
2. 2Curate. Delete or rewrite any test asserting intent instead of behaviour. Decision point: if the AI wrote assert "coupon works correctly"-style tests or invented behaviour (e.g., assumes extra keys raise — run it and see; v1's default ignores them), the test is characterizing the model's imagination, not the code. Execute every assertion against reality before keeping it.
3. 3Coverage gate.pytest --cov=pricing -q — push line coverage on models.py to ≥90% by asking the AI for tests targeting the uncovered lines only (paste the coverage report, not the whole file again).
4. 4Mutation gate — prove the tests bite. Run mutmut run --paths-to-mutate pricing/ then mutmut results. Target: ≥70% of mutants killed. Every surviving mutant is a behaviour your suite doesn't pin — paste survivors to the AI: "write one test that kills this mutant" — and re-run. Do not proceed to phase 2 below 70%.
5. 5Phase 2 — upgrade.pip install -U "pydantic>=2". Run pytest -q — and notice it does not burn: v2 ships deprecation shims, so most tests pass with warnings while a few fail on silently changed behaviour (this is the whole argument for phase 1 — no crash, wrong answers). Then run pytest -q -W error::DeprecationWarning to surface every shimmed v1 idiom as a hard failure (Pydantic's deprecation warnings subclass it — verified). Save both outputs as break-report.txt; together they are your migration worklist. Now run the agent loop: give your tool the failures and the module with "Migrate to Pydantic v2 idioms: field_validator, model_config / ConfigDict, model_dump, model_validate. Restore the v1 behaviours the characterization tests pin. Change ONLY what the upgrade requires. The characterization tests are the contract — make them pass without editing them." Apply, run, repeat until green with warnings-as-errors on. Decision point: if the agent proposes editing a characterization test, refuse unless you can prove the old behaviour was version-dependent (see Stuck for the one legitimate case).
6. 6Post-upgrade audit.grep -rn "validator\|parse_obj\|\.dict()\|class Config" pricing/ should return only v2-legitimate hits (field_validator contains "validator" — read the lines). Run the step-0 one-liner again: same output, new engine.
7. 7Mutation gate again. Re-run mutmut run once more on the migrated code — the suite should still kill ≥70%.

Verification

• •
  break-report.txt exists and shows real v1→v2 failures (proves tests were written before the upgrade).
• •
  pytest -q green on pydantic>=2 with characterization tests byte-identical to their phase-1 state (git diff tests/ empty across the upgrade commit, or the one documented exception).
• •
  Mutation score ≥70% both before and after migration (mutmut results outputs saved).
• •
  The step-0 one-liner prints the identical quote on v2.

Completion artifact

Keep the test suite, the upgrade diff (git diff of pricing/), break-report.txt, and both mutation reports. Self-grading rubric:

• Characterization honesty (0–3): every kept test asserts executed reality; no intent-tests survived curation = 3.
• Mutation rigor (0–2): ≥70% killed pre-migration, survivors triaged = 2.
• Contract preservation (0–2): tests unmodified through the upgrade (or one exception with written justification) = 2.
• Migration minimality (0–2): diff touches only v1 idioms — no drive-by refactors = 2.
• End-to-end proof (0–1): identical one-liner output on both versions = 1.

Pass = 7/10.

Stuck?

• mutmut can't find tests / hangs. Ensure tests/ sits beside pricing/ with a setup.cfg or run from the repo root; on stubborn setups, mutmut run --paths-to-mutate pricing/models.py narrows scope. Mutation runs are slow by nature — this module takes minutes, not hours.
• The agent "fixes" the coupon boundary during migration. Classic scope creep: v2 migration touches validators, and the model helpfully "corrects" >= 50. Your characterization test should catch it — that's it earning its keep. Revert, restate "only what the upgrade requires."
• One test legitimately can't survive: if you characterized v1's Optional[str] no-default behaviour (coupon being implicitly optional-with-None), v2 requires an explicit = None. Making that code change keeps the test green — the test still holds; only if you asserted on error message text must you loosen it, and that's the documented exception.

Client-safe variant

The two-phase discipline is the part clients care about. On restricted engagements: characterization tests can be generated by a local model (Ollama + coder model handles "write tests asserting current behaviour" well) or written by hand from AI-suggested case lists based on redacted signatures; mutmut and pytest are local OSS with no data egress. For the migration itself, deterministic tools do the mechanical 80% (bump-pydantic for this exact upgrade; OpenRewrite in Java land) — run the sanctioned LLM only on the residue that codemods leave behind, which is precisely the Google/Uber production pattern.

Answer key (don't read before attempting)

No seeded bugs — the "traps" are behavioural facts your characterization must capture; audit your suite against this list: (1) sku is stripped and uppercased; (2) qty 0 raises, but qty is not type-strict ("2" coerces — did you pin that?); (3) anystr_strip_whitespace strips customer_tier too — " gold " gets the discount in v1, and this is the sharpest trap: under v2 the renamed config key silently stops stripping, so " gold " loses its 10% — no crash, wrong money; only a characterization test catches it (verified); (4) the coupon boundary is >= 50, so subtotal exactly 50.0 stacks; (5) coupon: Optional[str] with no default is optional in v1 but required in v2 — omitting the key raises until the migration adds = None (the one legitimate code change from Stuck); (6) unknown extra keys are silently ignored by parse_obj in v1 — a suite asserting they raise was hallucinated, not characterized; (7) quote_total returns rounded floats and nested item dicts — shape drift under model_dump is what your dict-shape test exists to catch.