The same two steps as k-means — but soft.
k-means asks “which centroid is nearest?” and every point must pick one. A mixture model
asks “how likely is it that this point came from component j?” and the answer is
a set of fractions that add to 1, called the responsibilities.
E-step recomputes those fractions for every point; M-step refits each component
— its weight, its mean, and its covariance — using every point, weighted by how
much of it that component owns. That is the entire algorithm.
The two steps: Expectation and Maximization.
Things worth trying:
1. Overlapping, k = 2. Run it, then look at the band between the two blobs: those points
stay muddy for ever, because they genuinely are ambiguous and the model says so. Only 64% of
the points are ever more than 90% committed to a side — on Three blobs that figure is
100%. k-means has no way to express the difference: it must hand every one of them to a side.
This is the whole reason to use a mixture model: the uncertainty is part of the answer.
2. Elongated, k = 2. With Full the two ellipses tilt and lie along the cigars, and
every point lands in the right one. Now press Circular and re-run: forced to be round, the
components stop separating the two cigars and start cutting across them — only about
65% of points end up in the right group, and the log-likelihood falls from −1962 to
−2102. That is the punchline: k-means is a mixture model with circular, equal-sized
components and hard assignment. Everything GMM adds over k-means is in that one word,
covariance.