Data Concepts

Data Versioning

Pinning your code does not reproduce your run. If you cannot say which exact rows trained the model, then the comparison you just ran was not a comparison.

Two runs pin the same code and read two different datasets, so the difference between their scores cannot be attributed to the change anybody made

Why Does This Exist?

Two experiments, a week apart. Same repository, same commit pinned, one hyperparameter changed. The second scores 0.9 points higher, the change ships.

In between, 340 late-arriving events landed in the source table and 12 rows were corrected by a data-quality job. The second run trained on a different dataset, and nobody knows how much of that 0.9 came from the hyperparameter.

Pinning code doesn't reproduce a run. A model is a function of code and data, and the data half is usually the one that moves — because warehouse tables are mutable by design, and the query that defines your training set returns something different every time you run it.

That's the whole subject. Not tooling, not storage: the ability to answer "which rows trained this model" months later, without asking the person who built it.

Think of It Like This

A lab notebook that says 'the usual sample'

A chemist runs an experiment, gets a result worth publishing, and writes it up: apparatus, temperatures, timings, all exact. Under materials it says "the usual sample from the store cupboard".

Six months later nobody can reproduce it. The store cupboard has been restocked twice, a supplier changed, and one batch was contaminated. Every number in the write-up is correct and the experiment is gone.

The fix isn't a better notebook format. It's keeping a sealed, labelled portion of the actual sample, so a later reader can test the same material rather than a description of it.

Your warehouse is the store cupboard. SELECT * FROM events is "the usual sample".

How It Actually Works

What has to be pinned

Four things, and most teams do the first two.

Code — a commit. Configuration — hyperparameters, seeds, and the worker count, because that changes batch composition. Data — an identifier for the exact rows. Environment — library versions, since a tokeniser or a resampler changing behaviour between minor versions changes your inputs.

Three ways to pin the data

Immutable snapshots. Write the training set to storage as a dated, never-modified file set, and record the path. Crude, entirely reliable, and expensive at scale. It's also the only one that survives the source system being decommissioned.

Content hashing. Fingerprint the data itself: hash each row, sort the hashes, hash the concatenation. Two runs agree only if the bytes agree, and you get that check for free rather than trusting a path. Sorting the leaves first makes the hash independent of row order, which matters because a parallel export doesn't guarantee order. This is what tools like DVC and lakeFS give you, and you can implement the useful 90% of it in ten lines.

Point-in-time queries. Keep the source append-only, or use a table format that retains history — Delta, Iceberg, or a warehouse with time travel — and record the timestamp or table version instead of the data. Cheapest at scale, and it only works if nothing is ever updated in place. A dimension table overwritten nightly cannot answer "what did we know then", which is the same constraint the as-of join runs into.

Lineage is the other half

A version identifier tells you which data. Lineage tells you where it came from: which source tables, which transformation code, which upstream job run. Without it, a discovered problem in a source table can't be traced to the models that consumed it, so you can't answer "which of our deployed models needs retraining" — and that question arrives on a bad day.

The split has to be pinned too

Recording the dataset isn't enough on its own. The split is part of the experiment: a seed, or better, an explicit list of which ids landed in which fold. Two runs on identical data with different split seeds produce different scores, and on a small dataset the split noise can exceed the effect you're measuring — cross-validation schemes covers how large that is.

Best practice is to store the fold assignment as data rather than deriving it from a seed, because a library changing its shuffle implementation silently re-splits your data.

Worked example

Three rows, hashed. Then one late-arriving row lands, making four.

An order-independent content hash gives the same fingerprint for the three rows in any order — which is what you want, since export order isn't stable. Add the fourth row and the fingerprint changes completely.

That's the useful property: it's a single value you can log beside a metric, and it's impossible for it to match by accident. So a year later, "did these two runs use the same data" is a string comparison instead of an archaeology project.

Note what the hash does not tell you: what changed, or whether the change was legitimate. That's what validation and lineage are for. The hash only tells you honestly that something did.

Show Me the Code

1import hashlib2
3rows_a: list[str] = ["1,42.0", "2,17.5", "3,88.0"]4rows_b: list[str] = ["1,42.0", "2,17.5", "3,88.0", "4,9.5"]   # one late row landed5
6def dataset_hash(rows: list[str]) -> str:7    """Hash each row, sort the digests, hash the concatenation. Order-independent."""8    leaves = sorted(hashlib.sha256(r.encode()).hexdigest() for r in rows)9    return hashlib.sha256("".join(leaves).encode()).hexdigest()[:12]10
11print(len(dataset_hash(rows_a)))                           # -> 1212print(dataset_hash(rows_a) == dataset_hash(rows_a[::-1]))  # -> True   row order is irrelevant13print(dataset_hash(rows_a) == dataset_hash(rows_b))        # -> False  one extra row changes it14print(len(rows_a), len(rows_b))                            # -> 3 4

Twelve characters logged next to every metric, and "same data?" stops being a matter of opinion.

Watch Out For

A training query that reads a mutable table

SELECT * FROM events WHERE date >= '2026-01-01', run at training time, against a table that gets appended to and corrected continuously.

The query is a description of a dataset, not a dataset. Re-run it tomorrow and it returns something else — late events, corrected rows, a backfill someone ran, a soft-deleted record now filtered out. Every one of those is a legitimate improvement to the table and a change to your experiment.

The damage is specific: you can no longer attribute a score difference to anything. Two runs differ in code and data, so the effect you measured is confounded, and on a small effect the data drift can be larger than the change you shipped.

Materialise the query result before training, hash it, log the hash with the run, and keep the artefact. If storage cost makes that impossible, use point-in-time versioning and log the table version — but then the source must be genuinely append-only, and that's worth verifying rather than assuming.

Versioning the data and leaving the split to a seed

The dataset hash is logged, so the run looks reproducible. The split comes from random_state=42 inside a library call.

Two ways this breaks. The library changes its shuffle implementation in a minor release and the same seed produces a different partition, so your "reproduced" run trained on different rows. And a seed doesn't survive a change in row count: adding 340 rows re-shuffles everything, so the fold assignment for existing rows changes even though the seed didn't.

That second one is the quiet killer, because it means your dataset hash and your split can't both be stable unless the data is frozen.

Store the assignment as data. A two-column table of id and fold, versioned like any other dataset, hashed and logged. Then a re-run partitions identically regardless of library version or row count, and new rows have to be assigned deliberately instead of silently reshuffling the old ones.

The Quick Version

A model is a function of code and data. Pinning only the code reproduces neither.
Pin four things: code, configuration, data, environment. Library versions change tokenisers and resamplers.
Three ways to pin data: immutable snapshots, content hashing, point-in-time queries against an append-only source.
Sort the row digests before hashing, so the fingerprint doesn't depend on export order.
A table updated in place cannot answer "what did we know then", so point-in-time versioning needs append-only sources.
Lineage answers a different question from versioning: which source tables and jobs produced this, and therefore which models a bad source affects.
The split is part of the experiment. Store the fold assignment as versioned data, not as a seed.
A seed doesn't survive a change in row count: 340 new rows reshuffle every existing row's fold.
A twelve-character hash beside every metric turns "same data?" into a string comparison.