KL Divergence

Written $D_{\text{KL}}(p \parallel q) = \sum_i p_i \log(p_i / q_i)$, it is the extra surprise you accept by describing reality $p$ with your approximation $q$. It is zero exactly when the two distributions are identical, and positive otherwise.

It is not a distance, and the reason matters. $D_{\text{KL}}(p \parallel q)$ and $D_{\text{KL}}(q \parallel p)$ are different numbers, so the order of the arguments is a modelling decision. Putting the true distribution first penalises your model for assigning low probability where reality has mass, which produces an approximation that spreads out to cover everything. Reversing it produces one that concentrates on a single mode and ignores the rest — the difference between a blurry average and a confident guess.

Cross-entropy decomposes as $H(p) + D_{\text{KL}}(p \parallel q)$, and since $H(p)$ does not depend on your model, minimising cross-entropy and minimising KL divergence are the same optimisation. It also appears directly as the regularisation term in variational autoencoders and as the constraint keeping a fine-tuned policy near its reference model in RLHF and DPO.