Jensen-Shannon Divergence

KL divergence points one way and explodes on zeros. Jensen-Shannon meets in the middle: symmetric, always finite, capped at one bit.

Fair and 0.75-biased coins meet at their 0.625 midpoint, and the averaged KLs total 0.049 bits

Why Does This Exist?

$D_{KL}(P\\|Q)$ is directed and infinite when $Q$ misses $P$ 's support: useless for comparing disjoint generator outputs or text distributions with different vocabularies. Evaluation needs a symmetric, finite ruler that still respects information geometry.

JSD $= \\tfrac{1}{2}D_{KL}(P\\|M) + \\tfrac{1}{2}D_{KL}(Q\\|M)$ with $M = (P+Q)/2$ is that ruler: symmetric, in $[0, 1]$ bits ( $[0, \\ln 2]$ nats), zero exactly at equality, with $\\sqrt{\\mathrm{JSD}}$ a metric. It symmetrizes KL divergence for the comparisons in mutual information (MI is a KL; JSD is its tamed cousin).

The one idea you need first

KL $D_{KL}(P\\|Q) = \\sum P\\log(P/Q)$ prices extra bits using $Q$ where $P$ rules. The midpoint $M$ covers both supports, so neither KL explodes. Averaging removes direction.

Think of It Like This

Meeting halfway

Two hikers argue whose trailhead is better. Instead of pricing each from the other's start (KL, possibly impossible), both walk to the midpoint camp and price from there. Same camp both ways: symmetric, reachable, bounded effort.

Midpoint effort caps at the trail length (1 bit). Where it stops: halfway flattens real asymmetries. If direction matters (coding cost one way), keep KL.

How It Actually Works

Properties: $0 \\leq \\mathrm{JSD} \\leq 1$ bit, symmetric, finite for all pairs, $\\sqrt{\\mathrm{JSD}}$ satisfies triangle inequality. Identity: $I(X;Y)$ with $X$ a fair mixture bit equals $\\mathrm{JSD}(P\\|Q)$ : JSD is the information a sample reveals about its source. Original GANs minimize JSD implicitly; evaluation uses it for stability where KL would be infinite. Estimate via plug-in on shared support (midpoint covers union) with smoothing for small samples.

Worked example

$P = [0.5, 0.5]$ , $Q = [0.75, 0.25]$ , $M = [0.625, 0.375]$ . $D_{KL}(P\\|M) = 0.5\\log_2(0.5/0.625) \\times 2 = \\log_2(0.8) \\times -1$ ... compute: $0.5\\log_2(0.8) + 0.5\\log_2(1.3333) = 0.5(-0.3219) + 0.5(0.4150) = 0.04655$ . $D_{KL}(Q\\|M) = 0.75\\log_2(1.2) + 0.25\\log_2(0.6667) = 0.75(0.2630) + 0.25(-0.5850) = 0.1973 - 0.1462 = 0.05105$ . JSD $= (0.04655 + 0.05105)/2 \\approx 0.0488$ ... recompute precisely: first term $0.5 \\times -0.321928 = -0.160964$ ; second $0.5 \\times 0.415037 = 0.207519$ ; sum $0.046555$ . Second KL: $0.75 \\times 0.263034 = 0.197276$ ; $0.25 \\times -0.584963 = -0.146241$ ; sum $0.051035$ . Average $0.048795 \\approx 0.049$ bits. far under the 1-bit cap.

Code

1import numpy as np2
3p = np.array([0.5, 0.5])4q = np.array([0.75, 0.25])5m = 0.5 * (p + q)6jsd = 0.5 * (p * np.log2(p / m)).sum() + 0.5 * (q * np.log2(q / m)).sum()7print(round(float(jsd), 3))  # -> 0.049

Code 0.049 pins the hand arithmetic. Prose, program, and diagram agree.

Watch Out For

Reading JSD saturation as progress

Near-disjoint distributions all score ~1 bit: JSD saturates and its gradient vanishes. GAN and embedding comparisons stall with "everything maximally different" while shapes differ wildly.

Use Wasserstein or MMD for geometry among far-apart laws. If JSD sits at 0.99+ across variants, the ruler is pegged.

Comparing JSD across different support sizes

Vocabulary or binning changes move the 1-bit cap's meaning: finer bins inflate attainable JSD without real change. Cross-tokenizer JSD leaderboards compare rulers, not models.

Fix the support and smoothing first. If bins differ, renormalize or switch metric.

The Quick Version

JSD midpoints KL: symmetric, finite, capped at 1 bit, square-root metric.
Fair versus 0.75-biased coins score about 0.049 bits.
MI with a mixture bit is exactly a JSD; original GANs optimize it implicitly.
Saturates near 1 for disjoint laws; use geometric metrics there.
Fix support and smoothing before comparing across setups.
Keep directed KL when coding direction is the question.