Bayes' Theorem
The rule for updating a belief after seeing evidence, by combining how well the evidence fits with how plausible the belief was beforehand.
Think of It Like This
Like adjusting how likely you think it is raining outside after hearing thunder, combining your morning weather forecast with fresh evidence.
Bayes' theorem states . In words: the updated belief is the likelihood of the evidence times the prior belief, divided by how likely the evidence was overall.
Its most useful lesson is the base-rate trap, and it is worth doing the arithmetic once. A test that is 99% accurate for a disease affecting 1 in 10,000 people returns a positive result. Out of a million people, 100 have the disease and 99 test positive; the 999,900 healthy people produce about 9,999 false positives. So a positive result means roughly 99 in 10,098 — under 1%. The test is excellent and the answer is still "probably not", because the prior was so low.
In machine learning this is the difference between maximum likelihood and MAP estimation: multiplying in a prior is what turns a pure best-fit into a regularised one. L2 regularisation is a Gaussian prior, and Laplace smoothing is a prior that refuses to let any probability reach exactly zero.