docker build succeeds but takes ages, and docker images reports the new service
at 6.4 GB. Pushing it to the registry runs for nearly an hour on office broadband
and sometimes times out partway. A teammate's docker pull fills the free space on
her laptop, and the CI runner fails with "no space left on device". The service
inside works fine — the box around it is the problem.
A first containerisation of an ML service: FastAPI app, sklearn or similar model,
a repo that also holds a venv, a data folder and months of git history. The
Dockerfile was copied from a blog: FROM python:3.12 (full base) and a single
COPY . . before the install, with no .dockerignore anywhere. On the author's
machine nothing looks wrong — builds run, containers serve — until the image has
to travel: a registry push, a teammate's pull, a cloud deploy. Linked modules:
4.3 (Docker for ML), 4.1 (Notebook to repo).
Weigh the layers instead of guessing: docker history <image> prints one row per
layer with its size. The typical reading here: the full Python base contributes
about 1 GB, the pip layer what it must, and one COPY layer holds gigabytes it has
no business holding. COPY . . copies the entire build context, and with no
.dockerignore the context includes .venv/ (hundreds of MB of packages that pip
already installed into the image — now present twice), data/ (the customer csv,
which is a privacy incident travelling to every machine that pulls), and .git/
(every version of every file, forever). The build log confirms it up front:
"sending build context" reports hundreds of MB or more, when a clean context for
this repo is a few MB. Three separate mistakes stack: a heavy base, an unfiltered
context, and an indiscriminate COPY.
Three moves, then a re-measure. One: switch the base to python:3.12-slim — same
Python, minus compilers and tools the service never calls, roughly 900 MB saved.
Two: add a .dockerignore that excludes .venv/, .git/, data/,
__pycache__/ and models/, then re-admit exactly what ships with a negation
line such as !models/churn_pipeline.joblib. Three: replace COPY . . with
named COPY lines — requirements first, then code, then the one model file — so a
reviewer can see precisely what enters. Rebuild and verify: context down from
hundreds of MB to a few, docker images down from 6.4 GB to about 458 MB for the
churn service, push time from an hour to a couple of minutes. If the data file
already reached a registry, treat it as an exposure: pull access logs, rotate
anything sensitive, delete the tag — the 0.2 pushed-secret discipline applies.
Print docker images after every build and hold a written size budget (ours:
500 MB) — a number in the README beats an opinion in review. Prefer slim bases by
default and let the exception argue for itself. Write the .dockerignore in the
same commit that creates the Dockerfile, the .gitignore-first habit from 4.1
repeated at the container door. In review, read every COPY line and ask "what
exactly can this sweep in?" — COPY . . should trigger the same reflex as
git add . did in 4.1.
Q: "Your team's image is 6 GB and deploys are slow. Walk me through your
diagnosis and fix." Strong answer: start with docker history to weigh layers
rather than guessing, name the usual three culprits — full base, missing
.dockerignore, COPY . . — and give the expected sizes before and after (about
6 GB down to under 500 MB). Bonus signal: flagging a copied-in data file as a
privacy incident to be handled, not merely dead weight, and naming the size
budget as a standing check rather than a one-time cleanup.