Bayesian Linear Regression
Treating regression coefficients as probability distributions updated from a prior into a posterior as data arrives, so predictions come with honest, data-driven uncertainty.
With a normal prior and normal noise, the posterior over each coefficient is exactly normal too, and its variance provably shrinks as more data accumulates. Taking the posterior's peak value, called MAP estimation, with a normal prior produces the same point estimate as ridge regression — ridge's L2 penalty is a normal prior in disguise.
The real payoff over a point estimate is predictive uncertainty: a full distribution over plausible outcomes for a new input, not just one best-guess number.