Reproducibility in Machine Learning
Train the same code on the same data twice and you can get two slightly different models. Reproducibility is the discipline of locking down every source of randomness until the same inputs always give the same model.
Why Does This Exist?
Traditional software is deterministic. Compile the same source twice and you get the same binary. Train the same model code on the same data twice and you can get two slightly different models, and that difference poisons everything downstream. You cannot debug a production failure you cannot recreate. You cannot claim a new feature helped when the score gap might be random jitter.
Reproducibility is the engineering discipline of locking down every source of randomness until training is a pure function again: same code plus same data gives the same model, bit for bit where possible, within a stated tolerance everywhere else.
The one idea you need first. A random seed fixes the starting point of a pseudorandom number generator, so every shuffle, initialization and dropout mask that follows is identical across runs. Seeds are set once at the top of a script, before any randomness is consumed. The tooling that records which seed, code and data produced a run is experiment tracking, the closest neighbour of this page. This page locks the randomness down; distributed training spreads work across devices and multiplies every source of it; A/B testing is where irreproducible gains go to be exposed.
Think of It Like This
The same cake in two different kitchens
Two kitchens bake from the same recipe with the same ingredients. In one, the oven runs hotter on the left side. In the other, the cook stirs clockwise instead of counter-clockwise. The cakes come out slightly different, and nobody can say which step caused it. Reproducibility means shipping an identical kitchen along with the recipe: the same locked oven, the same stirring direction and speed, written down so precisely that both kitchens produce the same cake.
The recipe is the code, the ingredients are the data, the locked kitchen is a pinned container, and the stirring instructions are the random seeds.
It stops here: you can see an oven hotspot, while GPU nondeterminism comes from invisible timing differences between parallel cores. And cake tasting is a matter of opinion, while two model runs either share a weight hash or they do not. The kitchen story explains the locking. It understates how exact the check is.
How It Actually Works
Pin the environment
If your run used one library version and your colleague's used the next, the underlying math may have changed between them. Pin exact dependency versions, and for production runs train inside a version-controlled container, so the software kitchen is identical everywhere the code executes.
Seed every random number
Training consumes randomness constantly: weight initialization, data shuffling, dropout masks, augmentation. A serious seed block fixes all of them before anything else runs: the language's own generator, the operating-system hash seed, NumPy's generator, and the framework generators for CPU and every GPU. Miss one and its stream differs per run, quietly undoing the rest.
Tame the hardware
This is the hard one. A GPU sums thousands of values with thousands of cores racing, and whichever core finishes first contributes first. That would be harmless except floating-point addition is not associative: the order of summation changes the rounding, and tiny differences compound over hundreds of epochs. Frameworks ship deterministic kernel modes that fix the summation order, usually at the cost of speed. Exact bit-level matches can still break across GPU generations, so state the tolerance your comparisons use.
Worked example: when addition stops commuting
Take a toy decimal system that keeps three significant figures and rounds after every operation. Add , and in two orders.
Left to right: , which rounds to three figures as . Add the second and you get again, rounding to .
Right side first: exactly. Then , which rounds to three figures as .
Same three numbers, different summation order, different answer: against . Real GPUs keep far more digits than three, so each rounding is microscopic, but a training run performs billions of such additions, and the order changes run to run with core timing. That is the leak deterministic kernels close.
Watch Out For
Matching hashes on one GPU and mismatching on another
Everything seeded, deterministic kernels on, hashes identical across repeats on the same card. Then the same code runs on a newer generation card and the hash differs. Different architectures split and order parallel work differently, so bit-level equality across generations is not something seeding can promise. The symptom is exact matches within one machine type and small mismatches across types. Keep final production runs on one homogeneous set of cards, and compare across hardware within a tolerance instead of demanding equal hashes.
Seeding everything except the kernel picker
The seed block is in place and runs still jitter. The usual culprit is the convolution backend left free to benchmark kernels at startup and pick the fastest, a choice that varies with load and temperature. The symptom is run-to-run variation with identical seeds and identical code. Force deterministic algorithms on and benchmarking off, accept the slower run, and re-check the hashes before investigating anything else.
The Quick Version
- Reproducibility makes same code plus same data produce the same model, so failures can be debugged and gains can be trusted.
- Pin exact dependency versions and train production runs in version-controlled containers.
- Seed every generator before anything consumes randomness: language, OS hash, NumPy, and framework streams for CPU and each GPU.
- GPU reductions sum in timing-dependent order, and floating-point addition is not associative, so fix the summation order with deterministic kernels.
- Even fully seeded runs can differ across GPU generations. Compare within a tolerance there, and demand exact hashes only on identical hardware.