Maximum Likelihood Estimation

Pick the settings that make what you actually observed the least surprising thing that could have happened, which is precisely what training a model does.

After seeing seven heads in ten flips, every possible coin bias gets a score for how well it explains that result, and the estimate is whichever bias scores highest.

Why Does This Exist?

Training a neural network is maximum likelihood estimation. Not an analogy, the same procedure with a different function being optimized.

This page turns several separate looking loss functions into one idea. Cross-entropy for classification, mean squared error for regression, and the objective behind language model pretraining are all negative log-likelihoods of an assumed distribution. Once that clicks, the choice of loss stops being memorization and becomes derivation: write down the distribution you believe produced the data, take its negative log-likelihood, and the loss follows.

The practical payoff arrives when no standard loss fits. Counts, strictly positive values, and heavy tailed measurements have no lookup table entry, but the procedure still works. The prerequisite likelihood fixes the direction of the conditioning, and Bayesian inference shows what changes when a prior joins.

The one idea you need first

For observed data $D$ and parameters $\theta$ , the likelihood $P(D \mid \theta)$ asks how probable the fixed data is as $\theta$ varies. The data is known and frozen, while $\theta$ moves. A derivative then locates the peak by finding where the slope of the log-likelihood reaches zero.

Think of It Like This

Working out which bag the marbles came from

Someone draws ten marbles from one of three bags and shows seven red and three blue, without saying which bag was used.

Bag A is mostly blue, bag B is evenly mixed, and bag C is mostly red. The reasoning is immediate: bag A would rarely produce seven reds, bag B might occasionally, and bag C would do it often. So bag C is the answer.

That reasoning is maximum likelihood. There is no computation of which bag was probably chosen, only a comparison of how well each bag explains what was seen, with the best explanation winning. If bag C had been locked in a cupboard and rarely used, the answer might change, and that shift is what separates maximum likelihood from its Bayesian sibling. Plain maximum likelihood ignores prior plausibility entirely and looks only at fit.

How It Actually Works

Given data $D$ and parameters $\theta$ , the likelihood is the probability of the data under those parameters:

L(\theta) = P(D \mid \theta)

In plain words, $\theta$ names the candidate world and $L$ scores how well that world predicts the fixed observations. For $n$ independent observations the joint probability is a product:

L(\theta) = \prod_{i=1}^{n} P(x_i \mid \theta)

The estimate $\hat{\theta}$ is whichever $\theta$ makes that product largest.

Take the log, then negate

That product is unusable as written. A thousand probabilities near $0.1$ multiply to $10^{-1000}$ , which underflows to exactly zero in any float type, leaving a flat zero with nothing to optimize.

The logarithm fixes it by turning the product into a sum:

\log L(\theta) = \sum_{i=1}^{n} \log P(x_i \mid \theta)

In plain words, add log-probabilities instead of multiplying probabilities. The logarithm is strictly increasing, so the location of the maximum never moves, only its value. Sums near $-2.3$ a thousand times are perfectly well behaved. Then negate, because optimizers minimize:

\mathcal{L}(\theta) = -\sum_{i=1}^{n} \log P(x_i \mid \theta)

That is the negative log-likelihood, and it is the loss function. Every step here was mechanical.

Where the standard losses come from

Run the same procedure on three common assumptions:

Categorical for multi-class classification. Here $P(y_i \mid x_i, \theta)$ is the softmax probability of the true class, so the negative log-likelihood is $-\sum \log \hat{y}_{i,\text{true}}$ , exactly cross-entropy loss.
Bernoulli for binary classification. The negative log-likelihood expands to $-\sum [y \log \hat{y} + (1-y)\log(1-\hat{y})]$ , which is binary cross-entropy term for term.
Gaussian with fixed variance for regression. The Gaussian density carries $e^{-(y-\hat{y})^2 / 2\sigma^2}$ , so its log is $-(y - \hat{y})^2 / 2\sigma^2$ plus a constant. Negate, drop constants that do not affect the optimum, and $\sum (y - \hat{y})^2$ remains, which is mean squared error.

That last line deserves a pause. Mean squared error squares its errors because the Gaussian squares in its exponent, not for convenience and not merely for differentiability. Swap in a Laplace distribution and the same steps give mean absolute error, which tolerates outliers precisely because the Laplace has fatter tails.

Adding a prior gives regularization. Maximum likelihood asks only which $\theta$ explains the data best, so it happily picks extreme values for a slightly better fit. Maximizing $P(\theta \mid D) \propto P(D \mid \theta) P(\theta)$ instead gives MAP estimation, whose log form is the usual loss plus $-\log P(\theta)$ . A zero centred Gaussian prior becomes $\lambda \lVert \theta \rVert_2^2$ , which is L2, while a Laplace prior becomes $\lambda \lVert \theta \rVert_1$ , which is L1. The geometry of why the L1 spike produces exact zeros is in norms.

Worked example

Flip a coin ten times and observe seven heads. With $p$ as the unknown bias, each head contributes $p$ and each tail contributes $(1-p)$ :

L(p) = p^7 (1-p)^3

In plain words, this scores every candidate bias by how likely seven heads and three tails would be under it, which is the curve in the diagram above. Take the log:

\log L(p) = 7 \log p + 3 \log(1-p)

Set the derivative to zero:

\frac{d}{dp} \log L = \frac{7}{p} - \frac{3}{1-p} = 0

Multiply through by $p(1-p)$ to get $7(1-p) = 3p$ , so $7 = 10p$ and $\hat{p} = 0.7$ . The answer equals the observed proportion, but it was derived rather than assumed. The same derivation returns the sample mean for a Gaussian and the observed class frequencies for a categorical. Ten heads out of ten would give $\hat{p} = 1.0$ , declaring tails impossible on ten observations, which previews the overconfidence pitfall below.

Code

The grid search below lands on the same $0.70$ the calculus found, and the last two lines show why every implementation works in log space.

1import numpy as np2
3
4def neg_log_likelihood(p: float, heads: int, tails: int) -> float:5    # Sum of log-probabilities, never a product, so nothing underflows.6    return -(heads * np.log(p) + tails * np.log(1 - p))7
8
9grid = np.linspace(0.01, 0.99, 99)  # candidate values of p10nll = np.array([neg_log_likelihood(p, 7, 3) for p in grid])11print(round(float(grid[nll.argmin()]), 2))  # -> 0.712print(float(np.prod(np.full(1000, 0.1))))  # -> 0.013print(float(np.sum(np.log(np.full(1000, 0.1)))))  # -> -2302.59

The minimum sits at $0.70$ , matching $7/10$ . The raw product of a thousand $0.1$ values collapses to zero while the log sum stays finite near $-2302.59$ .

Watch Out For

Reading the likelihood backwards

$P(D \mid \theta)$ and $P(\theta \mid D)$ are different quantities, and swapping them is the most common conceptual error here.

The likelihood is the probability of the data given the parameters, not the probability that the parameters are correct. An estimate of $\hat{p} = 0.7$ does not mean there is a 70 percent chance the bias equals $0.7$ . It means $0.7$ explains the observed flips better than any other value does.

The consequence is concrete. A likelihood is not a distribution over $\theta$ and does not integrate to one over $\theta$ . Confidence intervals cannot be read off it directly, and likelihoods on different datasets cannot be compared. Model comparison scores like AIC exist precisely because raw likelihoods are not comparable across different parameter counts. Turning a likelihood into a belief about $\theta$ needs a prior and Bayes theorem.

Maximum likelihood assigning probability zero to the unseen

Maximum likelihood fits what it saw and only what it saw, so anything absent from training gets exactly zero probability, and zero probabilities are catastrophic rather than merely wrong.

A text classifier that never saw a word with some class sets $P(\text{word} \mid \text{class}) = 0$ , and because the model multiplies probabilities, one zero drives the whole document probability to zero. In log space the same failure is $-\infty$ , which then becomes nan on the backward pass. The ten heads out of ten coin is the minimal case: $\hat{p} = 1$ makes the next tail infinitely surprising.

The fix always has the same shape, which is a prior. Laplace smoothing adds one to every count, weight decay shrinks continuous weights, and label smoothing softens targets. All three refuse to let any probability reach exactly zero. When a model emits inf or nan on merely unusual data, check this first.

The Quick Version

Maximum likelihood picks the parameters under which the observed data is least surprising.
The likelihood $P(D \mid \theta)$ is a function of the parameters while the data stays fixed.
Products of many probabilities underflow, so work with the log, which never moves the maximum.
Negating the log-likelihood gives the loss function, which is where losses come from.
A categorical assumption gives cross-entropy and a fixed variance Gaussian gives mean squared error.
Seven heads in ten flips gives $\hat{p} = 0.7$ by $7(1-p) = 3p$ , matching the diagram peak.
$P(D \mid \theta)$ is not $P(\theta \mid D)$ , so a likelihood is not a belief about the parameters.
Anything unseen gets probability zero, which becomes $-\infty$ in log space, so smoothing is a prior.