Bayesian Nonparametrics and Inference

Bayesian nonparametrics lets the model grow its own complexity with the data, adding clusters and features as evidence arrives instead of fixing their count up front.

A fixed three cluster model stalls while a nonparametric model opens new tables as diners arrive

Why Does This Exist?

Parametric models fix complexity before seeing data: pick $k = 5$ clusters and live with it. Too few merges real groups, too many fits noise, and refitting per $k$ with model selection burns compute while answering the wrong question. Real datasets from cell types to topics grow new categories as collection continues.

Bayesian nonparametrics puts a prior on infinite partitions and lets the posterior decide how many components the data earn. Dirichlet process mixtures add tables as diners arrive, the Indian buffet adds dishes per diner, and consistency theory says when the posterior settles on truth. This overview maps that family; the process pages prove each construction.

The one idea you need first

A prior scores hypotheses before data, the likelihood scores data under each hypothesis, and the posterior is proportional to their product. A nonparametric prior spreads mass over unboundedly many components with weights that sum to one. Bayes theorem and Bayesian inference carry the finite case.

Think of It Like This

A restaurant that builds tables on demand

A fixed restaurant has five tables and turns away the sixth party. A nonparametric restaurant starts empty: the first diner sits alone, each arrival joins a busy table with probability proportional to its crowd or starts a fresh table with probability proportional to a concentration knob. Fifty diners might fill ten tables, and the hundredth may open an eleventh.

Tables are clusters, diners are data points, and the concentration knob tunes novelty appetite. Where the analogy stops: real posteriors also infer table parameters and merge lookalikes, while the restaurant story only assigns seats.

How It Actually Works

A Dirichlet process $DP(\alpha, H)$ draws discrete distributions: weights from stick-breaking (break a unit stick by Beta draws) and atoms from base $H$ . A DP mixture draws one such distribution, then samples each observation from a randomly picked atom's likelihood. The number of used components grows like $\alpha \log n$ , slowly and without bound.

Inference must handle the infinite tail. Gibbs samplers reassign diners table by table, variational methods truncate sticks at a large $K$ , and split-merge moves escape local partitions. Consistency asks whether the posterior concentrates on the true density or component count as $n$ grows; for densities yes under support conditions, for exact component counts often no without extra penalties.

Worked example

Seat 50 diners with concentration $\alpha = 5$ . The expected table count is about $\alpha \ln((\alpha + n)/\alpha) = 5 \ln(55/5) = 5 \ln 11 \approx 5 \times 2.398 = 11.99$ , roughly 10 to 12 tables. The 51st diner joins an occupied table with probability $50/55 \approx 0.909$ and opens a new one with $5/55 \approx 0.091$ . Growth is logarithmic: doubling to 100 diners adds only about $5 \ln 2 \approx 3.5$ tables.

Watch Out For

Reading tables as true categories

Posterior tables are a density device, not a census of real kinds. One skewed cluster often splits into two Gaussian tables, and the count drifts with $\alpha$ . Teams that report the posterior mode as the discovered number of cell types overclaim.

Validate clusters downstream or use a mixture-of-finite-mixtures prior when the count itself is the estimand.

Setting concentration by gut feel

Alpha of 0.1 freezes one table; alpha of 100 opens a table per diner. Both extremes fit training likelihood yet generalize badly. The symptom is a partition that looks absurd next to domain knowledge.

Place a hyperprior on $\alpha$ or cross-check table counts across a grid before trusting inference.

The Quick Version

Nonparametric priors span unbounded components and let data decide the used count.
Dirichlet processes build weights by stick-breaking and seat diners by popularity plus novelty.
Fifty diners at $\alpha = 5$ fill about ten tables, growing logarithmically after.
Density estimation is consistent under mild conditions; exact component counts need stronger priors.
Tables approximate densities, so do not report them as discovered natural kinds.