A segmentation ships with k=5 and the segments will not hold still. Rerunning the notebook moves hundreds of customers between clusters; two "different" segments have near-identical profiles; marketing keeps asking what separates segment 2 from segment 4 and nobody has an answer that survives a follow-up question. The number five itself has no defence beyond "that is what we used".
The k was inherited, not chosen: copied from a blog post, a previous project, or
a senior's old notebook, and never tested against this data. No elbow curve, no
silhouette score, no stability check across seeds — the evidence-gathering step
of module 1.9 was skipped entirely. Frequent wherever a clustering tutorial is
followed line by line, and wherever a deadline pushes a team straight from
"k-means works" to "ship the segments". The instability is often amplified by a
second skipped step: n_init=1 with no random_state, so even the same k lands
differently every run.
Linked module: 1.9 (Unsupervised learning).
When k is larger than the number of real groups, k-means must invent boundaries through the middle of genuine clusters, and those arbitrary cuts land somewhere new on every initialisation — that is the reshuffling. When k is smaller, real groups get welded together and the profiles blur. Confirm with the evidence that was never gathered: inertia and silhouette for k=2 through 9 on the scaled features. A silhouette that peaks well away from the shipped k, or sits flat and low around it, is the verdict. Then run the stability check: cluster twice with different seeds and crosstab the two labelings — labels that describe real structure agree almost everywhere; invented boundaries agree by coincidence. Finally read the near-twin profiles side by side: if two segments differ by less than the columns' spread, they are one group wearing two ids.
Choose k with evidence, on scaled features: elbow for the cost curve, silhouette
score for separation, a rerun-with-new-seed crosstab for stability, and a
business read of the named profiles for meaning. Ship the k where the evidence
agrees, and when the silhouette is flat across a range, prefer the smallest k in
that range that stakeholders can name and act on — fewer, explainable segments
beat many fragile ones. Set n_init=10 and a fixed random_state so the chosen
answer is at least reproducible while it is being judged.
House rule: every clustering deck carries its evidence page — the elbow and silhouette table, the stability crosstab, and one line per segment naming it from its centre. Any k that arrives without that page is a question, not a decision. Budget the check honestly: it costs minutes on a laptop, and it is the difference between segments marketing uses and segments marketing quietly ignores.
Q: "How do you choose k for k-means, and how do you know your segments are real?" Strong answer: name the three instruments — elbow, silhouette, stability across reruns — then the judgement layer: profiles a human can name and a flat silhouette treated as "weak structure", not as licence to pick any k. Bonus signal: mentioning that a k copied from another dataset carries no evidence for this one, and that fewer explainable segments beat more impressive-looking ones.