Feathers' rule for legacy code — pin current behavior with tests before changing anything — was always right and always skipped, because writing tests for code you don't understand is miserable. AI removes the misery: it drafts pin-down tests fast.
Tier: Core · Workflow area: Refactoring & Migration
The catch is a discipline inversion you must hold on to: characterization tests assert what the code does, not what it should do. Bugs get pinned too, on purpose, and mutation testing tells you when the net is actually tight enough to refactor over.
coverage.py, nyc, JaCoCo) and a mutation tool (mutmut, Stryker, PIT).Choose the seam and freeze the surface. Pick the public surface you will refactor behind (a class, a module's exported functions, an HTTP handler). Characterize at the seam, not the internals — internal tests weld the current structure in place and will all break when you refactor, which defeats the point.
Generate observation-first tests. The prompt must ban aspiration:
Write characterization tests for the public functions of legacy/pricing.py.
Rules:
- Assert CURRENT behavior only. If behavior looks like a bug (e.g., rounding
down where up is expected, silently swallowing invalid input), still pin
it exactly, and add a comment: # PINNED: possible bug, see notes.
- Do not "fix" anything by asserting intended behavior.
- Cover: typical inputs, empty/None/zero, boundary values, invalid input
(pin the actual exception type and message), and any branch on config.
- If you cannot determine an output without running the code, write the
test with assert result == TODO and list it — do not guess values.
The TODO rule matters: models guess expected outputs from what code looks like it does, which silently converts characterization into specification.
Fill TODOs by execution, not inspection. Run each TODO input against the real code and paste actual outputs back (pytest --tb=short output, or a quick REPL loop the assistant writes for you). Every assertion value must come from an execution, not from anyone's reading of the code — this is the step that makes them characterization tests at all.
Measure the net with coverage, then mutation.
coverage run -m pytest tests/characterization/ && coverage report --include="*/pricing.py"
mutmut run --paths-to-mutate src/legacy/pricing.py
Coverage says which lines execute; mutation score says whether the assertions would notice a change. Feed survivors back:
These mutants survived: [mutmut results — e.g., line 88: `<=` changed to `<`
not caught]. For each, write a test input that distinguishes original from
mutant, executing through the public seam. Tell me if any mutant is
genuinely unobservable at the seam.
Uber's test-generation work uses exactly this loop: generate, measure, regenerate against the gaps. Target high mutation score on the code you are about to change, not the whole file.
Log the pinned bugs, then refactor on the net. Collect every # PINNED: possible bug into tickets — decision needed later: fix (with a deliberate test change) or bless as behavior. Then refactor (or unleash the agent). The suite must stay green with zero test edits during the refactor: a red test means behavior changed; an edited test means you moved the goalposts. Bug fixes happen after, as separate commits that flip one pinned test each.
During the refactor, the characterization suite is read-only. The moment you edit a pinned test to make the refactor pass, you are no longer refactoring — you are changing behavior without deciding to.
src/ and write-enabled only on tests/ during this phase.ApprovalTests, Jest snapshots): for functions returning complex structures, snapshot the real outputs over a corpus of recorded inputs — fastest possible pinning, weakest at explaining what changed when it fails; pair with a few explicit assertions on the fields that matter.coverage.py, mutmut, Stryker, PIT) — the measurement loop runs identically everywhere.