Frequentist vs Bayesian

The Short Answer

The disagreement is about what a probability describes.

The frequentist view: a probability is a long-run frequency. Parameters are fixed, unknown constants, so it is meaningless to ask for "the probability that the weight is 0.7". You estimate the parameter and quantify how much your estimate would vary across repeated samples.

The Bayesian view: a probability is a degree of belief. Parameters are uncertain quantities with distributions, so you start from a prior, observe data, and use Bayes' theorem to get a posterior.

Here is the part that dissolves most of the argument: you are almost certainly using both already. Training a network by minimising a loss is maximum likelihood estimation, which is frequentist. Adding weight decay makes it MAP estimation with a Gaussian prior, which is Bayesian. Regularisation is the join, and nobody agonises over it.

Where They Differ

	Frequentist	Bayesian
A probability is	A long-run frequency	A degree of belief
Parameters are	Fixed and unknown	Random, with a distribution
Prior knowledge	❌ Not represented	✅ Explicit, as a prior
Output	A point estimate	A full posterior distribution
Core method	Maximum likelihood	Bayes' theorem
Uncertainty statement	Confidence interval	Credible interval
Interpretation of that interval	95% of such intervals cover the truth	95% probability the value is inside, given the data
Small-sample behaviour	Overconfident; can assign probability 0	Prior regularises it
Computational cost	Low — optimise once	High — sampling or variational approximation
In deep learning	The default (a loss to minimise)	Ensembles, MC dropout, variational inference
ML equivalent	Unregularised loss	Loss plus a regularisation term

Choose A When

Prefer frequentist methods when:

• You have a lot of data. The prior's influence washes out as the sample grows, so a point estimate is fine and much cheaper.
• You need one number to act on. A model has to produce a prediction; a posterior over predictions is often more than the downstream system can use.
• Cost matters. One optimisation run versus sampling a posterior is not a close comparison at scale, which is why essentially all production training is maximum likelihood with a penalty term.
• You are running an A/B test with a preregistered design. Fixed sample size, stated hypothesis, standard test. Well-understood, and the tooling is everywhere.

Choose B When

Prefer Bayesian methods when:

• Data is scarce and prior knowledge is real. With ten observations, maximum likelihood will happily conclude something is impossible because it did not happen — seven heads in seven flips gives probability 1. A prior prevents that, which is the same reason Laplace smoothing exists.
• Calibrated uncertainty is the deliverable. Medical triage, risk pricing, anything where "the model is unsure" must reach a human. A posterior gives you that; a softmax probability from an overconfident network does not.
• You need to know what the model does not know. Active learning and out-of-distribution detection both want epistemic uncertainty, which is a posterior-width question.
• Evidence arrives over time. Bayesian updating is sequential by construction — yesterday's posterior is today's prior. Frequentist tests generally assume a fixed sample size, and peeking at results inflates the error rate.

What People Get Wrong

Misreading a confidence interval. A 95% confidence interval does not mean a 95% probability the parameter lies inside it. It means the procedure produces intervals covering the truth 95% of the time. The reading people want is the Bayesian credible interval — which is one honest reason to use Bayesian methods when you need to explain the result.

Believing priors are the unscientific part. Every model encodes assumptions. A Bayesian prior states them where they can be argued with; a frequentist model's assumptions sit in the choice of likelihood, the loss, and the regularisation coefficient, unnamed. Explicit is not worse.

Thinking it is a lifelong allegiance. It is a per-problem tool choice, and most practical pipelines mix them without incident.

Confusing a softmax output with a posterior. A network's probability is a point estimate of a conditional distribution, produced by a model that is itself uncertain. It says nothing about parameter uncertainty, which is exactly why confidently wrong predictions are so common — see cross-entropy on why chasing zero loss destroys calibration.

Assuming Bayesian means slow and impractical. Deep ensembles and Monte Carlo dropout are cheap approximations that recover much of the benefit, and both are a small change to an existing training pipeline.