Reproducibility in MLOps: Same Model Twice

Reproducibility pins code, data, environment, and randomness so thoroughly that rebuilding months later produces byte-identical weights, turning debugging and audits from archaeology into replay.

Four pinned inputs, a code commit, a data snapshot hash, a container digest, and seed 42, rebuild into weights whose checksum matches the original exactly.

Why Does This Exist?

Traditional software is deterministic: same source, same binary. Machine learning adds three more ingredients that all vary silently: the data, the environment, and the randomness. Same code on new rows, under a new library version, with a fresh seed builds a different model with different behavior, and nobody can say why.

That breaks the two moments that matter most. A model misbehaves in production and the team cannot replay the run that built it. An auditor asks what data and logic produced a regulated decision and the answer is a shrug. Reproducibility fixes both by pinning all four inputs, so a rebuild is a replay. Runs get recorded through experiment tracking, datasets freeze via data versioning, the steps rerun under pipeline orchestration, and the full record lives as ML metadata.

The one idea you need first

A hash is a fingerprint for bytes: change anything and the fingerprint changes. Reproducibility treats hashes as proof. The data snapshot, the container image, and the final weights each carry one, and a rebuild passes only when every fingerprint matches.

Think of It Like This

A key cut at a locksmith

A good copy needs four things: the cut code, the right blank, a calibrated machine, and the same technique. Change the blank and the same code cuts a key that looks right and opens nothing. Let the machine drift out of calibration and two cuts from one code differ by a hair, which is enough to jam.

The mapping is direct. The cut code is your source commit. The blank is the data snapshot. The calibrated machine is the pinned container. The technique is the seed. The lock turning is the checksum match.

Where it stops: keys are cut metal, nearly perfectly repeatable. Training runs floating-point math across thousands of GPU threads whose finish order varies, so determinism here must be enforced against hardware that resists it, not assumed.

How It Actually Works

Four pins hold a run still. Miss one and the rebuild drifts.

The four pins

Code pins to a commit hash, so the exact training script is recoverable and no local edit goes unrecorded. Data pins to a snapshot hash, so later rows arriving upstream cannot leak in; a live query is not a pin, it is a hope. Environment pins to a container digest with exact dependency versions, because a minor CUDA or NumPy change shifts floating-point behavior across hundreds of epochs. Randomness pins to seeds set across every library before anything draws, covering initialization, shuffling, dropout, and sampling.

Worked example

A run ships with this card: code commit 9f3a2c1, data snapshot md5 8d2f44a1, image digest sha256:7c19e0b4, seed 42. Months later an engineer rebuilds from the card. The seed check comes first because it is cheapest: with seed 42 the first three uniform draws are 0.639, 0.025, and 0.275, and any other triple means the seeding order is wrong before a single GPU spins up.

Then the rebuild runs and the weights hash is compared: original 9f3a0c7d against rebuild 9f3a0c7d, equal, twelve-plus hex digits agreeing. That equality is the whole claim of this page in one comparison. If the data hash had differed by a single row, the weights would differ everywhere and the mismatch would point at the data pin, not the code.

The GPU caveat

Seeds alone do not tame graphics hardware. Parallel reductions finish in varying order and floating-point addition is not associative, so $(a + b) + c$ can differ from $a + (b + c)$ in the last bit, and those bits accumulate. Frameworks offer deterministic modes that serialize the risky ops. They cost speed, often visibly, which is exactly the trade: pay minutes per run for replayability, or pay weeks later when a failure cannot be replayed.

Code

The point of this snippet is the cheapest check in the discipline: seeds reproduce the same draws, and identical config bytes hash identically.

1import hashlib2import random3
4
5def seed_all(seed: int) -> None:6    """Pin the standard library generator before anything draws."""7    random.seed(seed)8
9
10seed_all(42)11draws = [round(random.random(), 3) for _ in range(3)]12print(draws)  # -> [0.639, 0.025, 0.275]13
14seed_all(42)15redrawn = [round(random.random(), 3) for _ in range(3)]16print(draws == redrawn)  # -> True17
18card = "git:9f3a2c1|data:md5:8d2f44a1|image:sha256:7c19e0b4|seed:42"19first = hashlib.sha256(card.encode()).hexdigest()20second = hashlib.sha256(card.encode()).hexdigest()21print(first == second)  # -> True22print(len(first))  # -> 64

Same seed, same draws, every time. Same card, same fingerprint. That is the mechanism in miniature.

Watch Out For

Seeded code that still drifts on GPU

Every generator is seeded, yet reruns disagree in the sixth decimal and diverge from there. The symptom is checksums that almost match, which sends people rechecking seeds that were never the problem.

The cause is parallel reduction order plus non-associative floating point. Fix it by enabling the framework's deterministic mode and disabling autotuning benchmark selection, then accept the slowdown as the price of replay. Verify with the weights hash, not with vibes.

Training straight from a live query

The pipeline selects from the production database at run time, so tomorrow's identical code reads tomorrow's rows. The symptom is a run card with no data hash and a rebuild nobody can even attempt.

Fix it by snapshotting inputs to immutable storage before training and recording the hash, or by using time-travel queries against a versioned table format pinned to the run timestamp. No hash, no reproducibility claim.

The Quick Version

Reproducibility pins four inputs: code commit, data snapshot, container digest, and every random seed.
A rebuild passes only when the weights checksum matches the original exactly.
Seed 42 gives draws 0.639, 0.025, and 0.275, the cheapest smoke test before spending GPU hours.
Deterministic GPU modes cost speed and buy replayability, which is the trade, stated plainly.
A pipeline reading live tables is not reproducible; snapshot the data and hash it, or drop the claim.