Gaussian Mixture Models
A Gaussian mixture models data as a weighted sum of several Gaussian components, giving soft, probabilistic clustering.
A density built from Gaussians
A Gaussian mixture model (GMM) represents a distribution as p(x) = sum_k pi_k N(x | mu_k, Sigma_k), where each component k has a mixing weight pi_k (summing to one), a mean mu_k, and a covariance Sigma_k. Any smooth density can be approximated arbitrarily well by enough components, which makes GMMs a flexible tool for clustering and density estimation.
Fitting with EM
Parameters are learned with the EM algorithm. The E-step assigns each point a responsibility r_nk, the posterior probability it came from component k. The M-step updates each mean as the responsibility-weighted average of the points, each covariance similarly, and each weight as the average responsibility.
# One EM iteration for a GMM (sketch)
resp = weights * gaussian_pdf(X, means, covs) # E-step
resp /= resp.sum(axis=1, keepdims=True)
Nk = resp.sum(axis=0) # M-step
means = (resp.T @ X) / Nk[:, None]
weights = Nk / len(X)Covariance structure and model choice
The covariance can be full, diagonal, spherical, or tied across components. Diagonal covariances scale to high dimensions but assume axis-aligned clusters. The number of components is chosen by validation likelihood or information criteria such as BIC, which penalizes extra parameters.
GMM versus k-means
k-means is the hard-assignment, equal-spherical-variance limit of a GMM. GMMs add soft membership and elliptical shapes, at the cost of fitting covariances and risking singular components when a cluster collapses; a small ridge added to each covariance prevents this. GMMs are widely used for speaker models, background subtraction, and as flexible priors.
- Soft assignments express uncertainty near boundaries
- BIC or held-out likelihood selects the component count
- Regularize covariances to avoid singular collapse