The gap between "works in my notebook" and "runs in the pipeline" is where DS projects go to die.
Tier: Core · Workflow area: Productionization
Agentic coding tools now execute the mechanical 80% of that refactor — module extraction, typing, tests, CLI, CI — in an afternoon, but only safely under review gates: an equivalence harness proving the package reproduces the notebook's numbers, and human review at each stage boundary. Heavy generation, mandatory review.
.ipynb.Skip full packaging for one-off analyses — over-engineering exploratory code is its own pathology.
preds.head(50)), feature matrix hash — to files. This is your equivalence oracle.Freeze the oracle. Add a final notebook cell writing oracle/metrics.json, oracle/predictions_sample.parquet, and oracle/feature_hash.txt (e.g. hashlib.sha256(pd.util.hash_pandas_object(X).values).hexdigest()). Commit these. Every refactor stage must reproduce them.
Plan with the agent before any edits:
Read notebook churn_v3.ipynb. Propose a package structure for production
(target: daily batch scoring via CLI). Constraints:
- src/churn/{data.py,features.py,train.py,predict.py,config.py}, tests/, cli
via argparse or typer, pyproject.toml, no notebook imports.
- Config (paths, dates, hyperparams) out of code into a yaml.
- Identify: hidden state from out-of-order cells, hardcoded paths, dead
cells, and anything nondeterministic (unseeded ops, set iteration).
Output the plan and the risk list. Do not write code yet.
Review the plan — especially the nondeterminism list, which is what usually breaks equivalence.
Refactor in gated stages, equivalence-checked at each. Stage order that works: (a) extract data loading + feature building; (b) extract training; (c) extract prediction/scoring; (d) add CLI + config; (e) tests + CI. After each stage, the agent must run the equivalence harness:
Stage (a) done? Run: python -m churn.features --config config.yaml, then
compare the feature matrix hash to oracle/feature_hash.txt and show me the
diff of predictions_sample against the oracle. If anything differs, stop
and report the cause — do not "fix" the oracle.
You review the diff and the code at each gate. Never let the agent proceed through two stages on green-it-said-so.
Make hidden state explicit. Notebooks accumulate order-dependent state (a cell run twice, a variable overwritten). Where equivalence breaks, the usual suspects: cell-execution order, implicit dtype changes, global random state consumed differently. The agent finds these fast when told to bisect: Binary-search the divergence: hash intermediates after each transform step in both code paths and find the first mismatch.
Generate the test suite with teeth (see pb-10 for depth): unit tests for each transform with tiny fixture frames, the as-of/leakage synthetic-row test if features are temporal, a smoke test running the full CLI on 1,000 sampled rows, and the equivalence test itself as a permanent regression test against committed oracle files.
CI and review. Agent writes the GitHub Actions workflow (lint, mypy on public interfaces, pytest, the smoke test). Then a real pull request with human review — review the diff, not the agent's summary of it. Anything you do not understand, have the agent explain, then verify the explanation against the code.
Deprecate the notebook. Replace its body with a header pointing at the package and a thin demo importing from src/. Two sources of truth immediately diverge.
An agent that cannot make the numbers match will eventually propose updating the oracle "since the new values look reasonable." That proposal is always a stop-and-investigate, never an accept.
pip install -e . && pytest && churn predict --config ... works on a clean machine/container.src/; config fully externalized (grep for hardcoded paths and dates).assert abs(a-b) < 1 passes everything. Tolerances should be tight (1e-9 for deterministic paths, documented looser bounds only where parallelism genuinely reorders reductions).jupyter nbconvert --clear-output before any assistant sees it.