Machine Epsilon

Machine epsilon is roughly $2.2 \times 10^{-16}$ for float64, $1.2 \times 10^{-7}$ for float32, and about $9.8 \times 10^{-4}$ for float16. It is the reason 0.1 + 0.2 == 0.3 is false, and it sets a hard floor on how many digits any computation can trust.

The consequence that costs the most debugging time is catastrophic cancellation. Subtracting two nearly equal numbers throws away their agreeing leading digits and promotes rounding noise into the result. This is why a numerical gradient check gets worse when you shrink the step size — truncation error falls as $h$ shrinks while cancellation error grows, so total error is U-shaped and the sweet spot for float64 is around $h \approx 10^{-5}$. It is also why the one-pass variance formula can return a negative number.

The same limit explains why comparisons use a tolerance rather than equality, and why the epsilon added inside a normalisation layer's denominator is there — not as a magic number, but to keep a division from meeting a value that rounded to zero.