Differential and Information Geometry

Curved spaces need their own calculus, and probability models form one such space where Fisher information plays the role of the ruler.

Probability models sit on a curved surface where Fisher distance replaces straight lines

Why Does This Exist?

Euclidean calculus assumes flat space: straight lines are shortest and parallel directions stay parallel. Model spaces violate both. Rotations, subspaces, and probability distributions each live on curved constraint sets where straight-line steps leave the set or mismeasure distance.

Differential geometry supplies the corrected toolkit (manifolds, tangents, metrics, geodesics), and information geometry applies it to statistics with the Fisher metric. Children: manifolds, Fisher metric, statistical manifolds. Optimizer sequel: natural gradient.

The one idea you need first

A manifold is a space that looks flat up close but curves globally, like the Earth's surface. At each point a flat tangent plane approximates it, and a metric defines local distances. See vectors, matrices, tensors for the linear background.

Think of It Like This

City blocks on a globe

Downtown, a street grid works: blocks are square and directions hold. Across a continent, the grid breaks: flight paths curve, parallels converge, and flat-map distances lie.

Differential geometry is the switch from street map to globe math. Information geometry notices that families of probability models form their own globe, with Fisher information as its latitude-longitude scale.

Where the analogy stops: statistical manifolds use divergence measures that are not symmetric distances, unlike miles on a globe.

How It Actually Works

A Riemannian manifold pairs each point $p$ with a tangent space $T_p$ and a positive-definite metric $g_p$ defining inner products there. Lengths integrate $\sqrt{g(\dot{\gamma}, \dot{\gamma})}$ along curves; geodesics minimize that length. For a parametric family $p(x; \theta)$ , the Fisher metric $g_{ij} = \mathbb{E}[\partial_i \log p \,\partial_j \log p]$ is the canonical choice, invariant to reparametrization. Amari's dual connections ( $\alpha$ -connections) then give projection theorems for fitting models.

Worked example

Bernoulli family, $\theta = p \in (0,1)$ , single trial:

Score: $\partial_p \log p^x(1-p)^{1-x} = x/p - (1-x)/(1-p)$ . Fisher: $\mathbb{E}[(\cdot)^2] = 1/(p(1-p))$ .
At $p = 0.5$ : metric $4$ . At $p = 0.1$ : metric $1/0.09 \approx 11.1$ . Same coordinate move $dp = 0.05$ covers Fisher distance $\sqrt{4}\times0.05 = 0.1$ near the middle but $\sqrt{11.1}\times0.05 \approx 0.167$ near the edge. Behavior changes faster at extremes, which Euclidean steps ignore.
Geodesic distance between $0.3$ and $0.7$ : $2|\arcsin\sqrt{0.7} - \arcsin\sqrt{0.3}| \approx 2|0.991 - 0.580| \approx 0.822$ , longer than the $0.4$ coordinate gap. Curvature stretches the path.

Watch Out For

Computing Euclidean averages of distributions

Averaging parameters of two Gaussians can leave the family or produce a nonsense covariance. The straight line between models is not a valid statistical path.

Use mixture or exponential geodesics (moment vs natural parameter interpolation) depending on which structure you must preserve.

Treating KL divergence as a distance

KL is not symmetric and fails the triangle inequality. Nearest-neighbor or clustering code built on it as a metric returns order-dependent answers.

Symmetrize (Jensen-Shannon) or use the Fisher-Rao distance when metric properties matter.

The Quick Version

Curved constraint sets need manifolds, tangent planes, and metrics instead of flat calculus.
The Fisher matrix is the canonical metric for probability families.
Geodesics give shortest behavior paths; divergences replace symmetric distances.
Coordinate gaps mislead near distribution boundaries where curvature peaks.
Amari duality yields projection theorems for model fitting.