Floating Point Arithmetic

What happens when you actually add and multiply rounded numbers: tiny errors that cancel, accumulate or explode depending on the order of operations.

Adding small numbers first keeps them alive while adding big first rounds them away, so summation order decides whether the total keeps its small change

Why Does This Exist?

Formats decide which numbers exist; arithmetic decides what happens when you compute with them. Every operation rounds its true result to the nearest representable neighbour, and those roundings interact: sometimes they cancel harmlessly, sometimes they accumulate into a wrong answer with a confident face. Floating point arithmetic is the study of that interaction: rounding modes, machine epsilon, catastrophic cancellation and summation order.

The two facts that organise everything: relative error per operation stays near machine epsilon ( $\epsilon \approx 2.2 \times 10^{-16}$ in double precision), but absolute error scales with operand size. So adding a tiny number to a huge one rounds the tiny one away entirely, while subtracting nearly equal numbers amplifies the rounding both already carried. Formats are taught in floating point formats; the resulting failure catalogue in numerical stability.

The one idea you need first

A double-precision number carries about 16 significant decimal digits. Machine epsilon $\epsilon = 2^{-52} \approx 2.2 \times 10^{-16}$ is the gap between 1 and the next double above it. Any operation on numbers near 1 rounds to multiples of $\epsilon$ ; near $10^{16}$ , consecutive doubles sit 2 apart and fractional parts stop existing.

Think of It Like This

A cash register that rounds every line

A register rounds each line item to the nearest dollar as it scans, then totals the rounded lines. A 49-cent pack of gum scanned beside a 2,000-dollar television contributes nothing: swallowed by the rounding of its line.

Scan gum with gum first and the cents accumulate into dollars before meeting the television, surviving the rounding. Same items, different order, different total. Catastrophic cancellation is the mirror image: two nearly equal totals subtracted, where the big matching dollars cancel and the total is built entirely from accumulated rounding dust.

How It Actually Works

Three mechanisms cover most incidents:

Absorption: $a + b$ rounds to $a$ when $|b| < \epsilon |a| / 2$ . In doubles, $10^{16} + 1 = 10^{16}$ , and $1 + 10^{-16} = 1$ . Small change dies beside big totals.
Catastrophic cancellation: $(a + \delta_1) - (a + \delta_2)$ cancels the $a$ 's and leaves rounding noise $\delta_1 - \delta_2$ promoted to the full answer. Relative error explodes even though every operation rounded correctly.
Summation order: adding numbers smallest-first (or pairwise, or with Kahan compensation) keeps small contributions alive. Error grows like $O(n\epsilon)$ ordered well, $O(\sqrt{n}\epsilon)$ typically random, and arbitrarily badly adversarially.

Worked example

The classic one-liner in double precision:

(0.1 + 0.2) - 0.3 = 5.55 \times 10^{-17} \ne 0

Neither 0.1 nor 0.2 exists in binary: they round to the nearest doubles, and those roundings ( $+5.5 \times 10^{-18}$ and $+1.1 \times 10^{-17}$ ) survive the addition, then survive the subtraction because 0.3 rounds slightly differently. The residue $5.55 \times 10^{-17}$ is pure rounding archaeology. Meanwhile $(1 + 10^{-16}) - 1 = 0.0$ : absorption kills the $10^{-16}$ before subtraction even sees it, since $10^{-16} < \epsilon/2 \approx 1.1 \times 10^{-16}$ .

Code

The point is the check: confirm both phenomena in three lines before designing any float comparison.

1print((0.1 + 0.2) - 0.3)  # -> 5.551115123125783e-172print((0.1 + 0.2) == 0.3)  # -> False3print((1 + 1e-16) - 1)  # -> 0.0

Equality on floats is almost always wrong; compare with a tolerance (abs(a - b) <= 1e-9), and add small numbers first when a sum must keep them.

Watch Out For

Testing float equality with ==

Loss comparisons, convergence checks and unit tests written as == on floats fail on rounding dust like the $5.55 \times 10^{-17}$ above. They pass on the author's machine and fail in CI under a different BLAS ordering. Use tolerances everywhere: relative for scale-free checks, absolute near zero, and never exact equality past arithmetic.

Shrinking finite-difference steps until the check breaks

Gradient checks subtract nearly equal function values, which is catastrophic cancellation by design. Below $h \approx 10^{-5}$ in doubles, rounding noise dominates the difference and the check fails on correct gradients. The error is U-shaped in $h$ : truncation falls while round-off rises, so park near the bottom and use relative tolerances.

The Quick Version

Every operation rounds; relative error stays near epsilon but absolute error scales with size.
$(0.1 + 0.2) - 0.3$ leaves $5.55 \times 10^{-17}$ : rounding residue, not a bug.
$1 + 10^{-16}$ absorbs to 1 in doubles: small change dies beside big totals.
Add smallest-first (or compensate) to keep small contributions alive in long sums.
Never compare floats with ==; gradient-check near $h = 10^{-5}$ , never far below.