Gaussian Mixture Models

Your data came from several bell curves mixed together. Fitting finds each curve, and every point gets a share of each one.

Two bell curves at 18.2 and 22.0 cm overlap at 20.5 cm, where weighted densities 0.0606 and 0.0547 give soft shares 0.53 and 0.47

Why Does This Exist?

A trawl survey measures 900 herring. Plot the lengths and you get a tall hump near 18 cm and a broader shoulder near 22 cm, young fish and older fish caught in one net. The count of young fish is the stock forecast, so the split matters.

Cut at 20.5 cm and tally, and a fish sitting exactly in the overlap is recorded as certain. That discards what you wanted: how unsure that assignment is, and how large each group really is once unsure fish count fractionally.

Two ideas make the rest click. A normal distribution is the bell shaped rule for measurements, set by a centre and a width. Likelihood is how probable your lengths are under a guess at those numbers, and fitting picks the guess that makes the observed lengths least surprising.

This page covers the soft split: fit several bell curves, give every point a share of each, and aggregate those shares. The hard split that forces one label lives in k-means. The density rule for points that fit nowhere lives in dbscan. How to judge any grouping once you have one lives in clustering evaluation.

Think of It Like This

Two spray cans on a fence

Someone stood at a fence with two paint cans. Pick a can, spray one dot, pick again, spray again, nine hundred times, choosing the first can about six times in ten. Each can throws an oval of mist, dense at the centre and thin at the edges, and the ovals overlap in the middle.

You arrive after they left. From the dots alone you can work out roughly where each can was held, how wide its mist was, and how often each was picked.

Now pick a dot from the overlap and ask which can made it. Nobody can. What you get is odds: the mist from the first can is slightly denser there, so the dot is a little more likely from that one, about 0.53 against 0.47.

Each can is one component, its position and mist width are the mean and variance, how often it was picked is the weight, and the odds on one dot are the responsibilities.

Where it stops: paint dots really did come from cans. Fish lengths did not come from bell curves. The mixture is a story you assumed to split the catch, and the shares are true inside that story, not in the sea.

How It Actually Works

Call the length $x$ . Call component $k$ one bell curve with centre $\mu_k$ , width $\sigma_k$ , and weight $\pi_k$ for how often it gets picked. Weights sum to one. The density at $x$ is the weighted sum:

p(x) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(x \mid \mu_k, \sigma_k)

$\mathcal{N}$ is the normal density, meaning the bell height at $x$ for that centre and width. The responsibility of component $k$ for one fish is its share of that sum:

r_k(x) = \frac{\pi_k \, \mathcal{N}(x \mid \mu_k, \sigma_k)}{p(x)}

Responsibilities sum to one per fish. They are the soft assignment.

You cannot maximise likelihood directly, because the assignments depend on the curves and the curves depend on the assignments. Expectation maximisation breaks the circle. The E step computes $r_k$ for every fish under current curves. The M step re-estimates each $\mu_k$ , $\sigma_k$ , and $\pi_k$ as averages over all 900 fish weighted by $r_k$ . A fish at 0.53 and 0.47 feeds both. Likelihood never falls across iterations, so the loop converges to a local maximum, which is why restarts matter exactly as they do for k-means.

Worked example: the overlap fish at 20.5 cm

Use the two curves from the diagram. Component 1 has $\mu_1 = 18.2$ , $\sigma_1 = 1.9$ , $\pi_1 = 0.60$ . Component 2 has $\mu_2 = 22.0$ , $\sigma_2 = 2.4$ , $\pi_2 = 0.40$ .

Weighted density at $x = 20.5$ for component 1 is $0.0606$ . For component 2 it is $0.0547$ . Add them:

p(20.5) = 0.0606 + 0.0547 = 0.1153

Divide each share by the total:

r_1 = 0.0606 / 0.1153 \approx 0.53, \qquad r_2 = 0.0547 / 0.1153 \approx 0.47

The fish is almost a coin flip. Peak heights tell the same story in reverse: component 1 peaks at $0.1260$ and component 2 at $0.0665$ , a ratio set by weight over width, not by drawing choice. The diagram draws both curves, the mixture above them, and the 20.5 cm line reporting 0.53 and 0.47.

Why this is not fuzzy k-means

Force every component spherical with equal width, force equal weights, then replace each $r_k$ with 1 for the largest and 0 for the rest. The two steps are now k-means. What you gain by not forcing it: full covariance lets a component stretch and tilt, unequal weights let a small year class stay small, and soft shares aggregate into fractional expected counts.

Choosing the count and the shape

Covariance type is the capacity knob. Full gives each component its own stretch and tilt at quadratic cost in column count. Diagonal allows different widths but no tilt. Spherical is one width. Tied shares one matrix. With forty columns and eight hundred rows, full overfits and diagonal is usually right.

Choosing $K$ is easier than in k-means because you own a likelihood. Fit $K$ from 1 to 8, score each with BIC, take the minimum. Still a heuristic and still assuming bell curves, but it beats squinting at an elbow.

Code

The point is the check: rebuild the 20.5 cm shares from the diagram numbers with plain arithmetic.

1import math2
3def weighted_density(x: float, mu: float, sd: float, w: float) -> float:4    z: float = (x - mu) / sd5    return w * math.exp(-0.5 * z * z) / (sd * math.sqrt(2.0 * math.pi))6
7d1: float = weighted_density(20.5, 18.2, 1.9, 0.60)8d2: float = weighted_density(20.5, 22.0, 2.4, 0.40)9print(f"{d1:.4f} {d2:.4f} {d1/(d1+d2):.2f}")10# -> 0.0605 0.0547 0.53

The printout gives 0.0605 and 0.0547, ratio 0.53, matching the hand computation. Change $x$ to 18.2 and the first share rises past 0.85. That movement is the softness working.

Watch Out For

A component collapsing onto a handful of points

The log likelihood climbs, then shoots up absurdly, and one component has eleven members with variance near zero. That is collapse. A bell curve narrowing onto near identical points sends its density toward infinity, and the optimiser is doing what you asked.

Symptom: one weight near zero with a tiny variance and a huge likelihood jump late in fitting. Fix: add a small floor to the covariance diagonal, the reg covar option in most libraries, and drop components whose weight falls under one percent. If it recurs with full covariance, switch to diagonal.

Reading responsibilities as real world chances

A report says a fish has a 53 percent chance of being young. It does not. It has responsibility 0.53 under a model that assumed lengths are a sum of two bell curves. If lengths are skewed or three year classes swam in, that number is precise and wrong.

Symptom: confident shares on data that fails a normality check inside each component. Fix: check whether each component members look bell shaped, refit with $K$ one higher and see whether the split survives, and where ground truth exists, check whether fish called 0.70 land that way about seventy percent of the time.

The Quick Version

A mixture claims each row came from one of $K$ bell curves picked with probability $\pi_k$ .
The E step gives every point a responsibility per component that sums to one.
The M step re-estimates each centre, width, and weight as responsibility weighted averages.
Our 20.5 cm fish draws weighted densities 0.0606 and 0.0547, so shares are 0.53 and 0.47.
K-means is the hard equal width equal weight special case of this loop.
Full covariance buys tilt, soft shares buy fractional counts, and BIC compares choices of $K$ .
A variance floor stops collapse, and responsibilities are model shares, not sea truth.