Multivariate Gaussian

The bell curve grown into many dimensions: a center plus a covariance matrix that stretches the cloud into a tilted ellipse.

Tilted one- and two-sigma ellipses around the mean show correlation 0.6 stretching the cloud, with center density 0.0995

Why Does This Exist?

Noise in ML is rarely one-dimensional. Sensor errors, embedding jitter, and weight priors all vary along many axes at once, and the variation leans: errors in $x$ predict errors in $y$ . A pile of independent one-dimensional bells cannot express that lean, so every model that reasons about joint uncertainty, from Kalman filters to Gaussian processes to variational posteriors, reaches for the distribution that can.

The multivariate Gaussian is that distribution, and it earns its ubiquity structurally. It is fully described by a mean vector and a covariance matrix. Its marginals and conditionals are Gaussian again, in closed form. Linear maps keep it Gaussian. And the central limit theorem makes it the default shape of averaged noise. The single-variable bell is probability distributions; the averaging behind covariance is expectation, variance and covariance.

The one idea you need first

A Gaussian in one dimension has a center $\mu$ and a spread $\sigma$ , with density falling exponentially in squared distance from center. In $d$ dimensions the center becomes a vector $\mu$ and the spread becomes a covariance matrix $\Sigma$ , while squared distance becomes the Mahalanobis form $(x - \mu)^T \Sigma^{-1} (x - \mu)$ . Everything below is that substitution, followed faithfully.

Think of It Like This

A spotlight on a tilted stage

A spotlight throws an elliptical pool of light: brightest at center, fading outward, stretched along one direction. Tilt the lamp and the ellipse rotates; widen the beam and it grows. The brightness at any floor tile is the density there.

The mean vector aims the lamp and the covariance matrix shapes the pool. Variances set the beam width along each axis, covariances tilt it, and the determinant sets the total light: a stretched pool spreads the same unit of probability thinner. Contours of equal brightness are ellipses, which is why Gaussian reasoning always draws ellipses, and why the diagram's rings are the whole story in miniature.

The analogy stops at the tails. Light has an edge, but Gaussian brightness never reaches zero, only fades. Real data with hard cutoffs needs truncated or bounded distributions instead.

How It Actually Works

The density in $d$ dimensions:

p(x) = \frac{1}{\sqrt{(2\pi)^d \det \Sigma}} \exp\left(-\tfrac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu)\right)

In plain words: center the point, measure its squared length in the covariance's own units (Mahalanobis distance), decay exponentially, and normalize by the matrix's determinant. Three structural facts run practice:

Marginals stay Gaussian. Drop coordinates and the rest is Gaussian with the corresponding sub-vector and sub-matrix. Reading one sensor of a Gaussian array never leaves the family.
Conditionals stay Gaussian. Fix some coordinates and the rest is Gaussian with a shifted mean and a shrunk covariance. Observing can only reduce uncertainty, never reshape its type.
Eigenvectors draw the ellipse. The covariance's eigenvectors point along the ellipse axes with lengths $\sqrt{\lambda_i}$ , which is why eigenvalues and eigenvectors are the natural sequel.

In plain words again: the mean says where, the eigenvectors say which way it stretches, the eigenvalues say how far, and every slice or projection keeps the same bell shape.

Worked example

Take $\mu = (0, 0)$ with $\Sigma = [[4, 1.2], [1.2, 1]]$ : standard deviations $2$ and $1$ with correlation $1.2 / (2 \times 1) = 0.6$ .

Determinant: $4 \times 1 - 1.2^2 = 2.56$ . Center density: $1 / (2\pi\sqrt{2.56}) = 1 / (2\pi \times 1.6) \approx 0.0995$ . Now the point $(2, 0)$ , one raw standard deviation along $x$ . Its Mahalanobis distance is not $1$ : with $\Sigma^{-1} = (1/2.56)[[1, -1.2], [-1.2, 4]]$ , the quadratic form is $4 / 2.56 = 1.5625$ , so the distance is $1.25$ . Correlation pushes the point effectively farther, because the cloud's tilt makes pure- $x$ excursions unusual. Its density is $0.0995 \times e^{-0.78125} \approx 0.0455$ , less than half the center's.

Code

The 2-by-2 density from scratch, no libraries beyond the standard one, reproducing the hand numbers.

1import math2
3cov = [[4.0, 1.2], [1.2, 1.0]]4det = cov[0][0] * cov[1][1] - cov[0][1] ** 25norm = 1.0 / (2 * math.pi * math.sqrt(det))6inv = [[cov[1][1] / det, -cov[0][1] / det], [-cov[1][0] / det, cov[0][0] / det]]7
8
9def density(x: float, y: float) -> float:10    quad = inv[0][0] * x**2 + 2 * inv[0][1] * x * y + inv[1][1] * y**211    return norm * math.exp(-0.5 * quad)12
13
14print(round(det, 4))                # -> 2.5615print(round(density(0.0, 0.0), 4))  # -> 0.099516print(round(density(2.0, 0.0), 4))  # -> 0.0455

The prints give 2.56, 0.0995, and 0.0455, matching the hand derivation exactly. The quad line is the Mahalanobis distance squared, the only part that changes with dimension.

Watch Out For

Measuring Gaussian distance in raw Euclidean units

One raw sigma along a correlated axis is not one sigma of surprise, as the worked 1.25 shows. Nearest-neighbor searches, anomaly thresholds, and confidence ellipses drawn in Euclidean distance misrank every tilted point. Whiten with the covariance first, or use Mahalanobis distance throughout, or the ellipse's tilt silently reorders your answers.

Assuming Gaussianity because only two moments were checked

Matching a mean and covariance proves nothing about shape: heavy-tailed, multimodal, and skewed distributions wear identical first-two-moment masks. Gaussian closures, Kalman updates, and analytic posteriors all break when the true noise is not bell-shaped. Check tails and slices before trusting the family's conveniences.

The Quick Version

The multivariate Gaussian is a mean vector plus a covariance matrix, and nothing else.
Its density decays exponentially in Mahalanobis distance, with the determinant normalizing.
Marginals and conditionals stay Gaussian in closed form; observing only shrinks uncertainty.
Eigenvectors aim the ellipse axes and eigenvalues set their lengths.
Correlation 0.6 makes one raw x-sigma worth 1.25 Mahalanobis, with density 0.0455 against 0.0995.