Kernel Density Estimation

Histograms jump with bin edges. KDE centers a smooth bump on every point and averages them into a curve with no bins to argue about.

Eight samples each contribute a dashed bump of width 0.3, and their average forms the smooth density with no bin edges

Why Does This Exist?

Histograms lie by binning: shift the edges and modes appear or vanish, and high dimensions make bins explode exponentially. Parametric fits lie by shape: data are rarely exactly Gaussian. Analysts need a smooth, bin-free picture of "where the data concentrate" before modeling.

KDE $\\hat f_h(x) = \\frac{1}{nh}\\sum_i K((x - X_i)/h)$ delivers it: one kernel $K$ per point, averaged, with bandwidth $h$ setting bump width. Gaussian kernels are the default. It needs only descriptive statistics as background and pairs with resampling in bootstrap and resampling.

The one idea you need first

A density integrates to 1 with tall where data crowd. Each kernel bump integrates to $1/n$ , so the average integrates to 1 automatically. Bandwidth is the only knob: small $h$ follows every point, large $h$ flattens everything.

Think of It Like This

Sand piles instead of buckets

A histogram sorts sand into buckets: move the bucket walls and the skyline changes. KDE drops one small sand pile on each grain and lets them merge into dunes.

Pile width is bandwidth: sugar-fine piles spike at every grain, gravel-wide piles flatten the desert. Where it stops: sand piles are physical and local, while KDE sums globally per query, which gets expensive on big $n$ without trees or FFT.

How It Actually Works

Bias shrinks as $h^2$ while variance shrinks as $1/(nh)$ : the classic tradeoff, minimized near $h \\sim n^{-1/5}$ in 1D with integrated squared error $\\sim n^{-4/5}$ . Silverman's rule $h = 0.9\\min(\\sigma, IQR/1.34)n^{-1/5}$ starts Gaussian-ish cases. Multivariate KDE needs full bandwidth matrices and degrades fast with dimension: effective samples shrink exponentially, so past ~5D prefer parametric or projection methods.

Worked example

Eight points $[1.0, 1.5, 2.0, 2.2, 3.0, 3.5, 5.0, 5.2]$ with Gaussian $h = 0.3$ . At $x = 2.1$ , kernel contributions peak for the 2.0 and 2.2 neighbors: each contributes $\\frac{1}{8}\\frac{1}{0.3\\sqrt{2\\pi}}\\exp(-0.5(0.1/0.3)^2) \\approx 0.157$ , distant 5.0 contributes under $0.001$ . Summing all eight gives $\\hat f(2.1) \\approx 0.42$ . With $h = 0.05$ the same $x$ spikes above 1.0 between points and crashes between them; with $h = 2$ it flattens near 0.12 everywhere.

Code

1import numpy as np2
3x = np.array([1.0, 1.5, 2.0, 2.2, 3.0, 3.5, 5.0, 5.2])4h = 0.35grid = np.array([2.1])6dens = np.exp(-0.5 * ((grid[:, None] - x) / h) ** 2).sum(axis=1)7dens = dens / (x.size * h * np.sqrt(2 * np.pi))8print(round(float(dens[0]), 2))  # -> 0.42

The hand sum and the code agree at 0.42. Change $h$ to 0.05 or 2.0 and watch spike versus flatten.

Watch Out For

Trusting default bandwidths blindly

Silverman assumes near-Gaussian data and oversmooths multimodality: two real humps merge into one, and analysts conclude one segment. Undersmoothed defaults do the reverse, inventing modes from noise.

Plot several bandwidths across a decade of $h$ . If the mode count changes, the default decided your conclusion, not the data.

KDE on high dimensions or bounded edges

In 10D, neighborhoods are empty and bumps never overlap: the curve is $n$ spikes on a plain. On bounded $[0, \\infty)$ data, symmetric kernels leak mass across the edge and understate near-zero density.

Cap KDE at low dimensions, use boundary correction or logs near edges, and switch families past ~5D. If most query points see one bump, dimensionality won.

The Quick Version

KDE averages one $1/n$ -mass bump per point; no bins, one bandwidth knob.
Bias $\\sim h^2$ , variance $\\sim 1/(nh)$ ; optimal $h \\sim n^{-1/5}$ in 1D.
Eight-point example at $h=0.3$ gives $\\hat f(2.1) \\approx 0.42$ .
Tiny $h$ spikes on noise, huge $h$ flattens real modes; scan a decade.
Past ~5 dimensions or on hard boundaries, correct or switch methods.
Naive queries cost $O(n)$ each; use trees or FFT at scale.