A dependency was upgraded to pick up a bug fix, the image rebuilt "successfully",
and the container still shows the bug. docker exec <ctr> pip show <package>
prints the old version, while the same pip command on the laptop shows the new
one. Rebuilding again changes nothing — the build even finishes suspiciously
fast, with CACHED printed against the install step. The code and the box have
quietly stopped agreeing about which library is inside.
A Dockerfile that grew by hand over weeks. Somewhere above the
COPY requirements.txt / RUN pip install -r requirements.txt pair, an earlier
line remains from a quick experiment: RUN pip install scikit-learn — unpinned,
typed straight into the Dockerfile months ago. The team upgraded the pin inside
requirements.txt and trusted the rebuild to pick it up. Docker builds on the
laptop reuse a local cache that has held that early layer since the day it was
written. Linked modules: 4.3 (Docker for ML), 4.1 (pinned requirements),
4.6 (CI builds).
A layer's cache key is the instruction text plus the exact bytes of the files it
copies. RUN pip install scikit-learn copies nothing and its text never changed,
so Docker replays the months-old layer — including the months-old package version
it downloaded back then. The cache does not know PyPI moved on; it only knows the
instruction looks the same. The changed requirements.txt did invalidate the later
install step, but pip found scikit-learn already present from the earlier layer
and left the old version standing (or the two lines fought, depending on pin
direction). Confirm in two moves: read the build log and note CACHED against the
suspect RUN line, then run docker build --no-cache once — if the fresh build
produces the correct version, the stale layer is proven. docker history shows
the layer's age in plain sight.
Make requirements.txt the single source of dependency truth: delete every ad-hoc
RUN pip install <package> line, keep the one
COPY requirements.txt + RUN pip install -r requirements.txt pair, and pin
every package with ==, the 4.1 rule. Rebuild with --no-cache once to flush
the poisoned layer, then verify inside the container:
docker exec <ctr> pip show scikit-learn must print the pinned version. Add
--pull to that clean build so the base image is refreshed too. From then on the
normal cached build is safe again, because the only install step re-runs exactly
when the pins change.
One install line, one pinned file — any second pip install in a Dockerfile is a
review flag. Understand the cache's contract and say it aloud in reviews: same
instruction text plus same copied bytes means replay, even when the outside world
has changed. Let CI build from a clean slate (fresh runners or scheduled
--no-cache --pull builds) so a poisoned local cache can never be the version of
record — the image CI ships is the truth, the laptop cache is a convenience.
After any dependency bump, verify inside the artifact
(docker exec ... pip show), not on the laptop.
Q: "You upgraded a library, rebuilt the image, and the container still runs the
old version. What happened?" Strong answer: explain the cache key — instruction
text plus copied file bytes — and how an unpinned RUN line replays a stale layer
forever because its text never changes. Name the confirmation steps (CACHED in
the build log, a --no-cache rebuild, docker history) and the durable fix:
single pinned requirements file as the only install source, clean builds in CI.
Bonus signal: noting that --no-cache is the diagnostic, not the daily habit —
the daily habit is a Dockerfile whose cache can be trusted.