Floating Point Formats

Every number format splits its bits between how large a value it can hold and how finely it can tell nearby ones apart, and training cares far more about range than fine detail.

Exponent bits buy range and mantissa bits buy precision, so bfloat16 keeps float32 range with its 8 exponent bits while float16 tops out at 65504

Why Does This Exist?

Two 16-bit formats use the same memory and bandwidth, yet one trains a large model reliably while the other needs an extra mechanism to avoid producing not-a-number.

The difference is the bit split. Float16 spends 5 bits on the exponent and 10 on the mantissa. Bfloat16 spends 8 and 7. That single choice decides whether small gradients survive and whether large activations overflow, which explains why loss scaling exists and why bfloat16 became the default where hardware supports it.

This page covers the arithmetic that does not change: range, precision, and the two failures. Which accelerator supports which format changes yearly, so check current hardware docs for support. How rounding interacts once values are stored is in floating point arithmetic, and the failure catalogue is in numerical stability.

The one idea you need first

A float is sign, exponent, and mantissa: $\text{value} = (-1)^{s} \times 1.m \times 2^{e - \text{bias}}$ . Exponent width sets range, the largest and smallest magnitudes. Mantissa width sets precision, how many significant bits each value carries. Range and precision trade directly because the bit budget is fixed.

Think of It Like This

Scientific notation with fixed boxes

You must write every number as $d.ddd \times 10^{ee}$ with a fixed number of boxes, some for digits and some for the exponent.

Spend boxes on the exponent and you can write $10^{38}$ and $10^{-38}$ , but nearby values collapse together. Spend them on digits and you separate 1.00001 from 1.00002 beautifully, yet $10^{38}$ no longer fits and overflows to infinity.

Gradients in a deep network span an enormous range, since a chain-rule product over many layers can land anywhere from $10^{-8}$ to $10^{5}$ , while their job is mostly direction, not the sixth digit. So spend the boxes on the exponent. That is exactly what bfloat16 does, and why it works where float16 struggles.

How It Actually Works

The formats that matter, with the two numbers that matter about each:

Float32, 8 exponent and 23 mantissa. Max $\approx 3.4 \times 10^{38}$ , epsilon $\approx 1.2 \times 10^{-7}$ or about 7 decimal digits. The baseline.
Float16, 5 exponent and 10 mantissa. Max 65,504, epsilon $\approx 9.8 \times 10^{-4}$ or about 3 digits. Smallest normal value $\approx 6.1 \times 10^{-5}$ .
Bfloat16, 8 exponent and 7 mantissa. Max $\approx 3.4 \times 10^{38}$ , identical range to float32, epsilon $\approx 7.8 \times 10^{-3}$ or about 2 digits.
TF32, 8 exponent and 10 mantissa. Float32 range with reduced precision, used inside matmul units while tensors stay float32.
FP8, two variants. E4M3 with 4 exponent and 3 mantissa and max 448 favours precision for forward activations. E5M2 with 5 exponent and 2 mantissa and max 57,344 favours range for gradients.

Float16 and bfloat16 are the same size and differ only in the 5/10 versus 8/7 split. That is the whole design decision.

Why range beats precision for gradients

Overflow is loud. Past the maximum comes infinity, then not-a-number, and the run dies. Float16 tops out at 65,504, which is genuinely low: a chain-rule product over 30 layers averaging 1.5 per layer reaches about $1.9 \times 10^{5}$ , straight past it.

Underflow is quiet and worse. A gradient of $10^{-8}$ sits far below float16 normal range, so it rounds to zero with no warning. Those parameters simply stop learning, and the plateau looks architectural. Bfloat16 shares float32 exponent width, so neither happens. It pays with about 2 digits of precision, which gradients tolerate because they indicate a direction.

Mixed precision is not casting the model. It keeps a float32 master copy of weights, scales loss so small gradients land inside narrow range, accumulates in float32 inside kernels, and keeps normalisation and softmax wider. Float16 needs loss scaling while bfloat16 usually does not, which is the clearest practical consequence of the range gap.

Worked example

Take $g = 10^{-8}$ and $g = 10^{5}$ , both of which real gradient distributions contain.

Float16 smallest normal is $2^{-14} \approx 6.1 \times 10^{-5}$ . Since $10^{-8}$ lies below even the subnormal floor near $6 \times 10^{-8}$ , it rounds to exactly zero, so that parameter receives no update. Bfloat16 smallest normal is about $1.18 \times 10^{-38}$ , so $10^{-8}$ is comfortably representable.

At the top end, $10^{5} = 100{,}000 > 65{,}504$ , so float16 overflows to infinity and the run ends one not-a-number later. Bfloat16 tops out at $3.4 \times 10^{38}$ , so it is unaffected. Loss scaling by $2^{15} = 32{,}768$ lifts $10^{-8}$ to $3.28 \times 10^{-4}$ , now inside float16 window, which is what the scaler buys.

The precision cost for contrast: bfloat16 epsilon is $7.8 \times 10^{-3}$ , so a $10^{-4}$ update to a weight of 1.0 vanishes without a float32 master copy. Small updates discarded silently stall training at worse loss for reasons no learning-rate sweep fixes.

Code

The point is the check: confirm range behaviour with standard-library packing only, since exponent width is the claim.

1import struct2
3
4def to_f16(x: float) -> float:5    return struct.unpack(">e", struct.pack(">e", x))[0]6
7
8def add_f16(a: float, b: float) -> float:9    return to_f16(to_f16(a) + to_f16(b))10
11
12print(to_f16(1e-8), to_f16(1e5 + 1e5))13# -> 0.0 inf14print(to_f16(1e-8 * 32768))15# -> 0.000327587127685546916print(add_f16(1.0, 1e-4))17# -> 1.0

Tiny $10^{-8}$ rounds to zero in float16 while scaled by $32{,}768$ it survives at $3.28 \times 10^{-4}$ , and $1.0 + 10^{-4}$ still returns 1.0. Those three lines are the range argument and the master-weight argument as executable checks.

Watch Out For

Switching to float16 without loss scaling and blaming the model

Calling half precision directly appears to work, then training diverges or plateaus for reasons that look architectural. Underflow to zero means some parameters stop updating while loss still falls on the rest, so the model trains to a worse optimum with no error. Overflow means infinity, then not-a-number, and the run dies at an arbitrary step.

Symptom: silent plateau or sudden not-a-number with no trigger. Fix: use the framework mixed-precision path with master weights and a scaler instead of casting by hand, log the scale value since repeated back-off means batches are being skipped, and watch the gradient norm since it drifts toward the ceiling first. Where hardware supports bfloat16, prefer it.

Comparing narrow-format runs without controlling seed variance

Reduced precision often shows up as wider spread across seeds rather than a worse mean, and a single run of each cannot see that. Rounding noise perturbs the trajectory on a non-convex surface, so different runs land in different places even when average quality is unchanged.

Symptom: one float16 run beats one float32 run, then the next loses. Fix: compare at least three seeds per configuration as mean and standard deviation, keep evaluation in the wider format, and treat any change smaller than seed spread as unmeasured.

The Quick Version

Exponent width sets range while mantissa width sets precision.
Float32 reaches $3.4 \times 10^{38}$ with epsilon $1.2 \times 10^{-7}$ ; float16 tops out at 65,504.
Bfloat16 keeps float32 range with an 8-bit exponent and pays with about 2 digits of precision.
Float16 zeroes $10^{-8}$ and overflows $10^{5}$ ; bfloat16 holds both.
Float16 needs loss scaling while bfloat16 usually does not.
Mixed precision means master weights, scaling, wide accumulation, and selective wider layers.
Without master weights a $10^{-4}$ update to 1.0 vanishes; compare narrow formats across seeds.