FP8 Training Is Not Free

Why Does This Exist?

When hardware vendors announced FP8 (8-bit floating point) support for model training, the pitch was irresistible: double your throughput and halve your memory footprint for free. Developers saw the benchmarks and immediately set their precision flags to fp8, expecting their training runs to finish twice as fast.

Then the loss curves exploded. Models failed to converge, gradients underflowed, and the promised speedup was swallowed by instability. We quickly learned that moving from 16-bit to 8-bit training isn't just a hardware trick—it's a numerical tightrope walk.

Think of It Like This

Imagine trying to paint a detailed masterpiece, but you are only allowed to use 8 distinct colors instead of 16 million. You can still paint the picture, but you have to be incredibly strategic about how you mix those 8 colors and where you use them. If you just grab colors at random, it turns into a muddy mess.

How It Actually Works

An FP8 number only gives you 8 bits of information. You have to divide those bits between the exponent (which controls the range of numbers you can represent) and the mantissa (which controls the precision).

Because the dynamic range of FP8 is so small, you cannot represent the massive values of activations and the tiny values of gradients at the same time. If a gradient is too small, it rounds down to exactly zero (underflow). If an activation is too large, it hits the maximum value and turns into infinity (overflow).

To fix this, modern FP8 training uses delayed scaling. Before a tensor is converted to FP8, the system calculates a scaling factor based on the maximum value seen in the previous training steps. It scales the numbers up so they fit perfectly into the FP8 range, performs the fast matrix multiplication, and then scales the result back down.

Watch Out For

The scaling operations cost compute. The hardware throughput number assumes you are just doing pure FP8 math, but in reality, you are spending time calculating scale factors, casting tensors between FP32 and FP8, and maintaining high-precision master weights. If your model is heavily bound by memory bandwidth rather than compute, the overhead of managing these scales can eat the entire FP8 speedup.

(Correct as of August 2026).

The Quick Version

FP8 training can double your speed, but only if you carefully manage scaling factors to prevent numerical underflow. It is not a free lunch; it requires sophisticated numerics work that often offsets the raw hardware throughput gains.

What to Read Next

To understand the mechanics of this, review post-training-quantization and gpu-utilization-and-profiling.