Pull requests merge while checks are still queued or red. Developers say "CI will catch up" or ping an admin to bypass the gate. Runs pile up in a queue during the day. A genuinely broken commit sits red on main for two days before anyone notices, because red stopped meaning anything weeks ago.
An ML team wired their full training pipeline into CI: every push re-downloads all packages (about 4 minutes, uncached) and retrains the model on the full dataset (about 33 minutes). Total: 40 minutes per push, on every push, including one-line README edits. Within two weeks the team's behaviour adapted around the gate instead of to it. Linked modules: 4.6 (CI/CD), 4.5 (fixtures and the frozen eval set), 4.4 (where retraining belongs).
A gate's protection equals how often it runs times how much people respect its verdict — and both collapse with latency. Past a few minutes, humans route around: they batch pushes, skip the wait, normalise bypassing. The 40 minutes decomposes with the run's own step timings: uncached installs (4 min) and a full retrain on real data (33 min) — neither of which verifies a promise about the code. The suite's actual promises (features correct, contract held, gate floor met on the frozen eval set) were designed in 4.5 to run on tiny seeded fixtures in seconds. CI was doing production training's job, badly, on every push.
Three cuts, in order of effort. (1) Cache dependencies: cache: "pip" on
setup-python keyed to requirements.txt — the churn repo's install fell from
46 s to 6 s. (2) Test on fixtures, not full retrains: unit, contract and data
tests run on 4.5's hand-typed twelve rows; the quality gate scores a small
frozen eval set with a fast seeded train. (3) Move heavy retraining out of the
push path entirely: a scheduled nightly workflow (on: schedule) retrains on
full data and alerts on regression. Target after the cuts: the push pipeline
under 5 minutes. Then re-arm the gate: branch protection back on, bypasses
revoked.
Watch pipeline duration like a metric, with a stated budget (five minutes is a good fresher-team default) and an alert when a run crosses it. Every new CI step must answer two questions: which promise does it verify, and what does it cost per push? Keep cheap steps first so failures arrive early. Anything that needs real data or real scale belongs in a scheduled job, not in the gate every push waits on.
Q: "Your CI takes 40 minutes and developers have started merging without it. What do you do?" Strong answer: name the behavioural law first — slow gates get bypassed, so speed is a correctness feature; then the measured breakdown from step timings; then the three-part fix (cache, fixtures over retrains, nightly schedule for heavy work) with a stated time budget. Weak answers punish the developers or buy bigger runners while leaving the retrain in the push path.