Fisher Information Metric

The Fisher matrix measures how sharply model behavior responds to parameters, giving the natural ruler for steps, bounds, and approximate posteriors.

Fisher ellipses stretch where parameters barely move predictions and shrink where they bite

Why Does This Exist?

Two parameter directions can look equal in coordinates while differing 100-fold in effect on predictions. Step sizes, confidence regions, and generalization measures built on Euclidean distance inherit that distortion. The Fisher matrix corrects it by scoring directions through the model's output distribution.

It also sets the fundamental estimation limit (Cramer-Rao) and drives natural-gradient optimization. Parent: information geometry overview. User: natural gradient.

The one idea you need first

The score $\nabla_\theta \log p(x;\theta)$ has mean zero under the model. Its covariance is the Fisher $F = \mathbb{E}[ss^T]$ . Steep log-likelihood slopes mean informative parameters.

Think of It Like This

A microphone array locating a sound

Each microphone hears the same clap with different clarity depending on placement. Directions with crisp timing pin the source down; muffled directions leave it vague.

Fisher information is that clarity map for parameters. Sharp directions get short confidence intervals and big natural-gradient corrections; dull directions get wide intervals and gentle treatment.

Where the analogy stops: microphones are fixed hardware while Fisher estimates shift with parameters and data batches.

How It Actually Works

$F_{ij} = \mathbb{E}_p[\partial_i \log p \,\partial_j \log p] = -\mathbb{E}_p[\partial_i\partial_j \log p]$ . Cramer-Rao: any unbiased estimator has $\text{Cov}(\hat{\theta}) \succeq F^{-1}/n$ . For Gaussians $\mathcal{N}(\mu, \Sigma)$ , the $\mu$ -block is $\Sigma^{-1}$ and the $\Sigma$ -block involves $\Sigma^{-1} \otimes \Sigma^{-1}$ : uncertain outputs flatten the metric.

Worked example

Bernoulli $p = 0.5$ vs $p = 0.9$ , $n = 100$ trials estimating $p$ :

Fisher per trial: $1/(p(1-p))$ . At $0.5$ : $4$ . At $0.9$ : $1/0.09 \approx 11.1$ .
Cramer-Rao floor: $\text{Var}(\hat{p}) \geq 1/(nF)$ . At $0.5$ : $1/400 = 0.0025$ (std $0.05$ ). At $0.9$ : $1/1111 \approx 0.0009$ (std $0.03$ ).
Sample proportion attains it asymptotically. Near edges each trial carries more information, so the same $n$ pins $p$ tighter. Euclidean confidence balls miss this entirely.

Watch Out For

Swapping empirical Fisher for true Fisher silently

Outer products of batch gradients (empirical Fisher) match the true Fisher only at the model distribution near optimum. Elsewhere ranks and scales differ and natural steps mislead.

Label which Fisher each formula uses and damp empirical versions harder.

Quoting Cramer-Rao for biased estimators

Regularized and deep estimators are biased, so beating the bound is legal and common. Claiming optimality from the bound alone is wrong.

State bias explicitly or use Bayesian and minimax variants of the bound.

The Quick Version

Fisher is the covariance of the score and the curvature of KL divergence.
It sets the Cramer-Rao floor on unbiased estimator variance.
Behavior-aware steps divide gradients by Fisher instead of trusting coordinates.
Empirical and true Fisher differ; damping and labels keep them straight.
K-FAC and diagonal forms make Fisher practical for large models.