Integration and Area Under a Curve

Why Does This Exist?

Every continuous probability you use is an integral. The bell curve means nothing until you learn that area under it between two points is the chance of landing there, and total area is exactly $1$. AUC spells it out: area under the curve. The hard term that makes variational autoencoders tricky is an integral. Expected values over continuous variables are integrals.

So this page exists because continuous probability needs it. Probability distributions says a density integrates to $1$. This is the page that makes that sentence mean something.

There is a second reason, and it is practical. Most integrals you meet in ML have no closed form, and recognising one is the skill. If you can spot the integral, see why it resists solving and reach for sampling, you can read a VAE paper. If not, you cannot.

Think of It Like This

How It Actually Works

Two ideas that turn out to be one idea.

Integration as accumulated area

Take $f$ from $a$ to $b$. Chop into $n$ slices of width $\Delta x$, freeze $f$ on each slice and make rectangles. The total is a Riemann sum:

Height times width, totalled. Thinner slices converge to the definite integral:

The sign is a picture of the sum. The $\int$ is an stretched S for sum, $f(x)$ is height, $dx$ is thin width. Two properties get used constantly: additivity $\int_a^c = \int_a^b + \int_b^c$, and area below the axis counts negative.

The antiderivative, and the theorem that connects them

An antiderivative $F$ of $f$ is any function with $F' = f$. Since $\frac{d}{dx}x^3/3 = x^2$, $x^3/3$ is an antiderivative of $x^2$.

The fundamental theorem says area is the change in any antiderivative:

Find a function whose slope is your curve, subtract its endpoint values. Infinitely many rectangles collapse to one subtraction. Constants cancel, so the $+C$ never touches a definite answer.

The rules you actually need

Three cover most ML uses. Powers: $\int x^n dx = x^{n+1}/(n+1) + C$ for $n \neq -1$. Exponential: $\int e^x dx = e^x + C$, which keeps Gaussians tractable. Reciprocal: $\int 1/x\, dx = \ln\lvert x \rvert + C$, where logs come from.

For several variables integrate one at a time while holding the rest fixed, the same freeze move behind derivatives and partial derivatives. A double integral over a region is an integral of an integral.

The honest part: most integrals cannot be solved

Differentiation is mechanical. Chain and product rules crank out any standard derivative.

Integration is not. Many ordinary functions have no elementary antiderivative, and $e^{-x^2}$ is the famous one. No formula gives bell-curve area between two points. This is proven, not missing knowledge, which is why tables existed and why norm.cdf is numerical.

That shapes practice more than any technique:

• Quadrature: clever Riemann placement. Great in one or two dimensions, hopeless in a thousand.
• Monte Carlo: random samples averaged. Error falls as $1/\sqrt{N}$ with no dimension term, so it is the only option in high dimensions and why sampling runs Bayesian ML.

Worked example

Integrate $f(x) = x^2$ from $0$ to $3$.

Antiderivative $F(x) = x^3/3$. Check: $\frac{d}{dx}x^3/3 = x^2$. Then:

Nine square units. Check against six right-edge slices of width $0.5$. Heights $0.25$, $1$, $2.25$, $4$, $6.25$ and $9$ total $22.75$, times $0.5$ gives $11.375$.

That overshoots $9$ because on a rising curve each right edge is the tallest point, so every box pokes above the curve. Width $0.25$ drops the estimate near $10.16$, width $0.01$ gives $9.07$. It converges slowly, error proportional to width, which is why plain right sums are never used for real work.

The diagram shades $f(x) = x^2/2$ and shows $4.5$, exactly half of $9$, same integral with the function halved.

Code

1import numpy as np2
3
4def riemann_right(f, a: float, b: float, n: int) -> float:5    dx = (b - a) / n6    x = a + dx * np.arange(1, n + 1)7    return float(np.sum(f(x) * dx))8
9
10f = lambda x: x**211print(round(riemann_right(f, 0, 3, 6), 4))  # -> 11.37512print(round(riemann_right(f, 0, 3, 300), 4))  # -> 9.045113print(round(riemann_right(f, 0, 3, 30000), 4))  # -> 9.000514rng = np.random.default_rng(0)15xs = rng.uniform(0, 3, size=100000)16print(round(float(np.mean(f(xs)) * 3), 2))  # -> 9.0 approx

Exact is $9$ and both routes approach it. Grids die in high dimensions while Monte Carlo only needs samples.

Watch Out For

The Quick Version

• A definite integral is the limit of a Riemann sum: heights times widths as widths go to zero.
• It is signed, below axis subtracts, and additive over adjacent intervals.
• An antiderivative is a function whose derivative is your function.
• The fundamental theorem says $\int_a^b f = F(b) - F(a)$. Infinite boxes collapse to one subtraction.
• Integration undoes differentiation, which is why calculus is one subject.
• Three rules cover most ML needs: powers, $e^x$ and $1/x$ giving $\ln\lvert x \rvert$.
• Most integrals have no closed form. The Gaussian provably does not, so norm.cdf is numerical.
• Grid quadrature costs $k^d$ and fails past a few dimensions. Monte Carlo is $1/\sqrt{N}$ in any dimension.

What to Read Next

• Jacobian and Hessian
• Kullback–Leibler Divergence