The very first CI run on an ML repo fails with FileNotFoundError: models/churn_pipeline.joblib (or a missing csv, or a missing .env), while
pytest -q prints all green on your laptop — and on every teammate's laptop.
The team's first reaction is to distrust CI: "the pipeline is misconfigured,
the code is fine."
First week of wiring GitHub Actions onto an existing ML repo. The model file is gitignored — correctly, per module 4.1's ruling that binaries stay out of history. Every laptop trained the model weeks ago, so every laptop passes. Nobody has run the tests on a machine that started empty. Linked modules: 4.6 (CI/CD with GitHub Actions), 4.1 (Notebook to repo), 4.5 (Testing ML code).
A runner is a fresh clone: it holds exactly what git ls-files lists and
nothing else. Laptops accumulate untracked state — trained models, generated
data files, .env files, packages installed but never pinned. The gap is
provable in two commands: git ls-files models/ prints nothing, while
ls models/ on your laptop shows the joblib. Any test that loads that file
passes locally and must fail in CI. The general law: "works on my machine"
means "depends on something unrecorded". CI did not break the repo; it revealed
that the repo was never complete on its own.
Make the pipeline create what the tests need. For a fast, seeded repo like the
churn project, add python train.py as a step before pytest — the runner
builds the joblib in seconds, and the gate tests find it where config.MODEL_PATH
points. For heavy models, train once in one job and pass the file forward with
actions/upload-artifact / actions/download-artifact, or pull a versioned
model from a registry (module 4.4's home for exactly this). Do NOT "fix" it by
committing the joblib: binaries bloat every future clone and go stale silently —
the 4.1 ruling stands.
Treat CI as the definition of working: red in CI means broken, whatever any laptop says. Run 4.1's teammate-clone ritual (fresh folder, fresh venv, train, test) before wiring CI — it catches this class of bug for free. For every file a test loads, be able to point at the workflow step or artifact that produces it. Keep models travelling as artifacts or registry entries, never as commits.
Q: "Your tests pass locally but CI fails with a missing model file. Walk me
through your diagnosis and fix." Strong answer: state the fresh-clone principle
in one sentence, give the git ls-files vs ls evidence pair, offer the two
real fixes (create the file in CI, or hand it forward as an artifact / registry
pull), and explicitly reject committing the binary — naming why. Mentioning
that CI is the team's ground truth, not a nuisance, signals production
experience.