On Tuesday a churn model posts the best recall the team has seen, and the number goes into a stakeholder deck. On Friday the review asks for the model behind it — and nobody can produce it. Retraining with "the usual settings" scores lower every time. A joblib file from that week exists, but nobody can say which settings, which feature list or which data produced it. The author is certain it was real; the terminal that proved it has been closed for three days.
A tuning phase run by hand: many python train.py invocations with edited settings,
scores read off the terminal, the "best" noted mentally or in a chat message. No
tracking was in place — no run records, no logged params, no data fingerprint. Very
common in week one of any tuning effort, and in teams that moved from notebooks to
scripts (4.1) but stopped there. Linked module: 4.4 (MLflow tracking & registry).
The run existed once, in two fragile places: process memory and terminal scrollback.
Both are gone. Work backwards through what a record would have held. Params: unknown —
the winning C, feature list and split settings were edited in place and overwritten
by later attempts. Code state: unknown — uncommitted edits meant no commit marks the
moment. Data: unknown — the CSV was regenerated during the week, so even a perfect
settings match may score differently. Confirm the hole quickly: git log around
Tuesday shows no commit, there is no mlruns folder, and the joblib file's timestamp
is the only metadata it carries. A score with no linked record cannot be
distinguished from a lucky split, an edited feature list, or a copy-paste error in
the deck.
Stop re-running from memory — guesses burn hours and prove nothing. First, mark the Tuesday number as unverified wherever it was shared; an unreproducible score is a claim, not a result. Second, instrument the training script the 4.4 way: start_run wrapping the fit, log_params, log_metrics, log_model, plus a data hash and the git commit as tags. Third, re-run the sweep as recorded runs and let the runs table say what the settings can actually deliver. If a recorded run matches the Tuesday number, register that model and move on; if none does, the honest score is the best recorded one.
Adopt the rule that names this entry: a run that left no record did not happen. Recording must be done by the script, not the human — instrument once, and every future run files itself. Log the four rebuild facts on every run: params, metrics, data hash, git commit. Commit before sweeping, so every run points at real code. Anything shown to a stakeholder must first be a registered model version, because a registry version keeps its back-link to the run forever.
Q: "A teammate's Tuesday model was the best ever, and by Friday nobody could rebuild it. What went wrong, and what do you change?" Strong answer: name the real failure — the run existed only in memory and scrollback, so params, code state and data are all unrecoverable — then give the systemic fix: script-enforced tracking with params, metrics, data hash and commit on every run, and a registry for anything shared. The strongest candidates add the integrity point: an unreproducible score must be publicly downgraded to "unverified", not quietly defended.