A repo that has run for months breaks on a fresh machine. pip install -r requirements.txt finishes without errors, then the code fails at runtime:
TypeError: got an unexpected keyword argument, an AttributeError on something
that "was always there", or a deprecation warning that became a hard error.
git log shows no relevant code change. The old machines still work; only fresh
installs are broken.
The requirements file lists names without versions — pandas, scikit-learn,
joblib — written during a notebook-to-repo restructure and never revisited.
Everyone who installed early got compatible versions and moved on. Six months
later a new joiner sets up, or a CI runner rebuilds, or a server is re-imaged —
and pip resolves "latest of everything". Real ecosystem examples of the breakage:
scikit-learn renamed OneHotEncoder(sparse=...) to sparse_output= and later
removed the old name; numpy 2.0 deleted long-deprecated aliases. Linked modules:
4.1 (Notebook to repo), 0.1 (Your machine, set up right).
An unpinned requirement means "whatever version exists on install day", so the
repo describes a different environment every month while pretending to describe
one. The code did not break; the environment underneath it moved. Confirm by
diffing environments, not code: run pip freeze on a working machine and on the
broken one and compare — the version columns will disagree exactly where the
traceback points. The absence of a code diff is itself the evidence: when nothing
in git changed and behaviour changed, the unrecorded half of the project — the
environment — is the suspect.
Recover the known-good environment, then write it down. On a machine that still
works, read the true versions with pip freeze, and pin the direct dependencies
in requirements.txt with == — pandas==2.2.3, not pandas. The broken
machine reinstalls from the pinned file and comes back. If no working machine
survives, pin to versions released around the repo's last-known-good date and walk
forward one package at a time until the clone-test ritual passes. Commit the pins;
the fix is not real until a fresh venv on a fresh machine reproduces the numbers.
Pin from day one: a hand-curated requirements file naming only what the code
imports, every line with an exact version — plus a full pip freeze snapshot kept
as a deploy record. Upgrades happen on purpose, not by accident: a periodic pull
request bumps the pins, runs the tests (4.5) and the 4.1 clone-test, and gets
reviewed like any other change. The rule to remember: "latest" is not a version,
it is a different environment every day.
Q: "A service broke after a redeploy. The team swears nothing changed. Where do
you look?" Strong answer: state that the artifact is code plus environment, and if
the code diff is empty, diff the environment — pip freeze from the working and
broken hosts side by side. Name unpinned requirements as the usual cause, pinning
as the immediate fix, and scheduled upgrade pull requests as the long-term
discipline. Bonus signal: a concrete example, like an argument renamed between
library versions, shows you have lived through one.