Integrals in Probability

For anything measured on a continuous scale, probability is area rather than height — which is why the chance of hitting an exact value is zero and why so many useful quantities have no formula.

For a continuous variable the height of the curve is not a probability; only the shaded area between two points is, which is why the chance of landing on any exact value is zero

Why Does This Exist?

Three tools you use constantly are integrals, and none makes sense without this page.

norm.cdf is an integral. Every p-value, confidence interval and z-score lookup is area under a bell curve up to some point. There is no algebraic formula for it, which is why that function is a numerical routine and why statistics tables existed for a century.

The denominator in Bayes theorem is an integral. For continuous parameters the evidence term needs integrating over every possible value. That integral is why Bayesian methods are hard and why sampling and variational methods exist. The discrete version and its priors live in probability distributions.

The VAE loss is a workaround for an integral. The data probability under the model needs integrating over the whole latent space. It cannot be done, so training optimises a lower bound instead. The integral is step one, the bound is step three.

Think of It Like This

Asking how much a single point on a map weighs

You hold a map shaded by population density, people per square kilometre. Dark cities, pale countryside.

Ask how many people live in this province and the map answers: multiply density by area and total it. Ask how many live at these exact coordinates and the answer is zero. A point has no area, so however dark the shading, it contains nobody.

That is not a defect. It is what a density means. The shading is people per unit area, not people, and you get people only by multiplying by an area. A very dark point says the region around it is crowded, nothing about the point itself.

A probability density works the same. Its height is probability per unit of $x$ , not probability. Multiply by a width and you get probability. That is why a density can exceed $1$ while a probability cannot.

How It Actually Works

For a discrete variable every rule is a sum. For a continuous one every sum becomes an integral. That is the whole translation.

Density to probability

A density $p(x)$ meets two conditions:

p(x) \ge 0 \quad \text{and} \quad \int_{-\infty}^{\infty} p(x)\, dx = 1

Never negative, total area exactly one. Probability comes from intervals:

P(a \le X \le b) = \int_a^b p(x)\, dx

Two consequences surprise people. First, $P(X = c) = 0$ for any single $c$ , because zero width has zero area, so $\le$ and $<$ are interchangeable here. Second, $p(x)$ can exceed $1$ . A Gaussian with $\sigma = 0.1$ peaks near $3.99$ . Nothing is wrong, it is a rate and narrowness compensates.

The CDF is the integral of the density

The cumulative function accumulates area from the left:

F(x) = \int_{-\infty}^{x} p(t)\, dt

The chance of landing at or below $x$ . It runs $0$ to $1$ and never falls. Differentiate it and you get the density back, so CDF and PDF are an antiderivative pair. That is the most useful fact here, and it is why practical work happens in CDF space: any interval is one subtraction $F(b) - F(a)$ , and every p-value is a CDF call.

Expectation becomes an integral

Discrete expectation sums $x$ times probability. Continuous version:

E[X] = \int_{-\infty}^{\infty} x\, p(x)\, dx

Each value weighted by density, totalled across the line. More generally $E[g(X)] = \int g(x) p(x)\, dx$ , the form that appears everywhere because a loss is an expectation of per-example cost. Variance follows the same swap with $(x - E[X])^2$ inside.

The normalising constant, and where things go wrong

Often you can write a shape proportional to a density but not the density itself:

p(x) = \frac{1}{Z} \tilde{p}(x), \qquad Z = \int \tilde{p}(x)\, dx

$Z$ forces total area to $1$ . Discrete $Z$ is a sum you can compute, exactly what a softmax denominator is. Continuous $Z$ is an integral, and that is where tractability dies.

Bayes with parameters $\theta$ and data $D$ shows it:

P(\theta \mid D) = \frac{P(D \mid \theta)\, P(\theta)}{\int P(D \mid \theta)\, P(\theta)\, d\theta}

Numerator is easy, both pieces chosen by you. Denominator integrates over every setting, hopeless for millions of parameters. Every practical method dodges it: conjugate priors make it solvable for a few families, MAP maximises the numerator and drops $Z$ , MCMC samples without knowing $Z$ , variational methods bound it.

Worked example

Uniform on $[0, 2]$ . Width $2$ needs height $p(x) = 0.5$ for area $1$ .

P(0 \le X \le 1) = \int_0^1 0.5\, dx = 0.5

E[X] = \int_0^2 x \cdot 0.5\, dx = 0.5 \left[\frac{x^2}{2}\right]_0^2 = 1

Midpoint, as symmetry demands. Narrow it to $[0, 0.5]$ and $p(x) = 2$ keeps area $1$ . The density is $2$ , above $1$ , yet every probability stays below $1$ because widths are small.

Gaussian case: standard normal $P(-1 \le X \le 1) \approx 0.6827$ , the familiar $68$ percent, and it is an area. No elementary formula exists for $\int e^{-x^2/2} dx$ , which is why the diagram labels that region $0.68$ instead of deriving it.

Code

This snippet checks the uniform arithmetic and the Gaussian area with only numpy and the standard library.

1import math2import numpy as np3
4
5def uniform_pdf(x: np.ndarray, lo: float, hi: float) -> np.ndarray:6    return np.where((x >= lo) & (x <= hi), 1.0 / (hi - lo), 0.0)7
8
9grid = np.linspace(0, 2, 200001)10pdf = uniform_pdf(grid, 0, 2)11print(round(float(np.trapezoid(pdf, grid)), 4))  # -> 1.012print(round(float(np.trapezoid(grid * pdf, grid)), 4))  # -> 1.013print(float(uniform_pdf(np.array([0.25]), 0, 0.5)[0]))  # -> 2.014print(round(0.5 * (1 + math.erf(1 / math.sqrt(2))), 4))  # -> 0.8413

The first two prints confirm total area $1$ and mean $1$ . The third shows density $2$ is fine. The last is $\Phi(1)$ , so $\Phi(1) - \Phi(-1)$ is $2 \times 0.8413 - 1 \approx 0.6827$ .

Watch Out For

Comparing densities across different units or parameterisations

A density shifts under a change of variable by a Jacobian factor, because the widths you multiply by changed. Probabilities stay identical, density values do not. So log-likelihoods on different preprocessings can read $-2.3$ versus $-4.1$ for the same model on the same data.

Compare continuous likelihoods and bits-per-dimension only across identical preprocessing. A positive continuous log-likelihood is also fine, since density above $1$ has positive log.

Assuming a normalising constant can be computed

The common failure is writing an unnormalised density, getting a working loss, and never noticing $Z$ was dropped. Dropping it is correct for ranking or MAP, where $Z$ has no gradient. It fails the moment you need a real probability, a calibrated confidence, a likelihood comparison or an energy model.

Know whether your objective needs $Z$ . If you quote a probability, name the approximation that produced it.

The Quick Version

For continuous variables every probability rule becomes an integral instead of a sum.
A density is probability per unit, not probability. It may exceed $1$ .
$P(X = c) = 0$ always, because zero width has zero area.
The CDF is the integral of the density, differentiating it returns the density.
Expectation is $\int x\, p(x)\, dx$ , and $E[g(X)] = \int g(x) p(x)\, dx$ covers every loss.
The normalising constant forces total area to $1$ . Discrete is a sum, continuous is often intractable.
Bayes denominator is that integral, which is why MAP, MCMC and variational methods exist.
Densities shift under reparameterisation, so compare likelihoods only on identical preprocessing.