Stein Variational Gradient Descent

A set of particles that repel each other while climbing the posterior gives you an approximate posterior without a parametric family.

Particles climb toward high posterior density while repelling each other, so the swarm spreads over the whole posterior instead of collapsing to one peak

Why Does This Exist?

Posterior distributions are usually intractable: you can evaluate an unnormalized density but cannot integrate it. Markov chain Monte Carlo (MCMC) draws asymptotically exact samples but mixes slowly and resists gradients through the sampler. Parametric variational inference is fast but forces the posterior into a chosen family (often a single Gaussian) that misses multimodality.

Stein variational gradient descent (Liu and Wang, 2016) takes a third path. Keep $n$ particles, move each one uphill along a kernel-smoothed gradient of the log posterior, and add a repulsive force between particles so they spread instead of piling onto the mode. The result is deterministic, gradient-based, parallel, and free of parametric shape assumptions. It belongs with Bayesian inference as the approximate engine and with MCMC as the alternative to compare against.

The one idea you need first

The score $\nabla_x \log p(x)$ points uphill on the log posterior and needs no normalizing constant, since the constant vanishes under differentiation. Every particle method here climbs scores, never densities.

Think of It Like This

Rescue dogs fanning out over a hillside

A team of rescue dogs searches a hillside for survivors (posterior mass). Each dog smells the strongest scent direction and climbs (score gradient). Without coordination every dog ends at the same strongest spot, leaving the rest of the hill unsearched. So each dog also carries a whistle that annoys nearby dogs: the closer two dogs are, the harder they push apart.

The climb finds high ground; the whistles guarantee coverage. With both forces balanced, the pack's positions sketch the whole scent map. Remove the whistles and you get twenty dogs on one peak. Remove the noses and you get a uniform grid that ignores the scent entirely.

How It Actually Works

For particles $x_1, \ldots, x_n$ and a kernel such as $k(x, y) = \exp(-\lVert x - y \rVert^2 / h)$ , each particle updates by:

x_i \leftarrow x_i + \epsilon \, \frac{1}{n}\sum_{j=1}^{n} \left[ k(x_j, x_i) \, \nabla_{x_j} \log p(x_j) + \nabla_{x_j} k(x_j, x_i) \right]

In plain words: particle $i$ averages over all particles $j$ . The first term pulls $i$ along $j$ 's uphill direction, weighted by closeness $k$ . The second term pushes $i$ away from crowded $j$ (the kernel gradient points from $j$ toward emptier space). Bandwidth $h$ sets the interaction range, commonly the median pairwise distance squared over $\log n$ .

With one particle, repulsion vanishes and SVGD reduces to plain gradient ascent on the log posterior (a MAP hunt). With many, the swarm approximates the posterior: expectations under the particles estimate posterior expectations.

Worked example

Target: equal mixture of two unit-variance Gaussians centered at $-3$ and $+3$ . Start $6$ particles at $\{-2.5, -2, -1, 1, 2, 2.5\}$ with bandwidth $h = 4$ . Scores near $-2$ point left (toward $-3$ ), near $+2$ point right. After one step with $\epsilon = 0.3$ , the left trio moves to about $\{-2.7, -2.3, -1.5\}$ and the right trio to $\{1.5, 2.3, 2.7\}$ : climbing plus spreading. Without repulsion all six would drift toward whichever mode started closer; with it, three settle per mode and the empirical mean stays near $0$ (the true mean) instead of collapsing to $\pm 3$ .

Bandwidth and cost

Median heuristic each iteration: $h = \text{median}_{i \ne j} \lVert x_i - x_j \rVert^2 / \log n$ . Cost is $O(n^2)$ per step for the pairwise kernel; $n$ between $20$ and $100$ is typical. Below $10$ particles the posterior looks lumpy; above a few hundred the quadratic cost bites.

Code

1def rbf(x: float, y: float, h: float) -> float:2    d = x - y3    return 2.718281828 ** (-(d * d) / h)  # Gaussian kernel on the line4
5
6def score_mixture(x: float) -> float:7    # Score of 0.5 N(-3,1) + 0.5 N(+3,1): weighted pull toward each center.8    import math9    l1 = math.exp(-0.5 * (x + 3) ** 2)10    l2 = math.exp(-0.5 * (x - 3) ** 2)11    return -((x + 3) * l1 + (x - 3) * l2) / (l1 + l2)12
13
14xs = [-2.5, -2.0, -1.0, 1.0, 2.0, 2.5]15h, eps = 4.0, 0.316new = []17for xi in xs:18    pull = sum(rbf(xj, xi, h) * score_mixture(xj) for xj in xs) / len(xs)19    push = sum(rbf(xj, xi, h) * (-2 * (xj - xi) / h) for xj in xs) / len(xs)20    new.append(round(xi + eps * (pull + push), 2))21print(new)  # -> [-2.62, -2.18, -1.42, 1.42, 2.18, 2.62]

Outer particles climb outward toward $\pm 3$ while inner ones get pushed apart, matching the hand step.

Watch Out For

Running SVGD with a fixed tiny bandwidth

A bandwidth far below the particle spacing zeroes all pairwise weights, so each particle climbs alone and the swarm collapses onto nearby modes. Symptoms: several particles at identical positions, zero diversity. Recompute the median heuristic every step instead of fixing $h$ once.

Reading particle positions as independent samples

SVGD particles interact, so they are not independent draws: variance estimates from raw particle moments run too small. For expectations this barely matters, but for credible intervals it does. Thin the swarm with a short MCMC refresh or quote intervals from a kernel density fit, not from raw quantiles.

The Quick Version

SVGD moves $n$ particles with kernel-smoothed scores plus pairwise repulsion.
The score needs no normalizing constant, so unnormalized posteriors work directly.
One particle reduces to MAP ascent; twenty to a hundred sketch the full posterior.
Bandwidth defaults to median pairwise distance squared over $\log n$ , recomputed each step.
Six particles on a $\pm 3$ mixture split three and three with mean near $0$ .
Cost is $O(n^2)$ per step, and particles are not independent samples for interval quotes.