Data Versioning

Why Does This Exist?

Code alone does not reproduce a model. A model is code plus rows, and rows move. Warehouse tables accept late events, corrections, and backfills by design, so the same query next week returns different bytes.

A team that versions code perfectly but overwrites training files in place loses the past. A score drop cannot be blamed on code or data because both changed at once. Audits fail for the same reason: nobody can name the exact rows behind a shipped model. This page covers row identity and restore. Run identity lives in experiment tracking, run repeatability lives in reproducibility, and field meaning lives in ML metadata.

Basics first: a hash is a short fingerprint of bytes where any byte change remakes the fingerprint. A pointer file is a tiny text file with that fingerprint plus a path. Git stores the pointer well and stores multi-gigabyte tables badly, so the split is the whole design.

Think of It Like This

How It Actually Works

Split the job: Git tracks history, remote storage tracks bytes, and the hash joins them.

Hash the bytes

Read each file, fingerprint the content, and combine file prints in sorted order so export order cannot change the result. One changed pixel, one corrected cell, or one added row remakes the top hash. Store objects under the hash as the name, so equal bytes share one object and different bytes never collide in practice.

Store once remotely

Upload the bytes to object storage under the hash name. Never overwrite. A 3-row snapshot stays at 04cb forever. A 4-row snapshot lands beside it at 8c40. Old models keep reading old bytes while new models read new ones.

Commit the pointer with code

Write the hash, row count, and source query into a small pointer file and commit it with the model code. Cloning gives history without weight. A checkout command reads the pointer at that commit, fetches the matching bytes, and places them locally. Code state and data state move in lockstep through one link.

Worked example: 3 rows become 4

Start with 3 rows: 1,42.0, 2,17.5, and 3,88.0. Hash each row, sort the prints, and hash the join, keeping 4 hex chars as the short pointer. That pointer reads 04cb in the diagram.

Add one late row: 4,9.5. The set is now 4 rows. The same recipe now reads 8c40. The pointer changed completely from one added row:

One row added, one new object stored, zero old bytes touched. Restoring the v1 commit fetches 04cb with 3 rows. Restoring v2 fetches 8c40 with 4 rows. Order does not matter because leaves sort first, so a reshuffled export still matches 04cb while any content change breaks it.

Lineage is the other half

Identity says which rows. Lineage says where they came from: source tables, transform code, and job run. Without it a bad source cannot be traced to the models it fed, so the retrain list cannot be built on a bad day.

Code

The point is the fingerprint: hash rows in sorted leaf order so content decides the pointer, with the same 3 versus 4 rows as the worked example.

1import hashlib2
3
4def short_hash(rows: list[str]) -> str:5    leaves = sorted(hashlib.sha256(r.encode()).hexdigest() for r in rows)6    return hashlib.sha256("".join(leaves).encode()).hexdigest()[:4]7
8
9rows_v1: list[str] = ["1,42.0", "2,17.5", "3,88.0"]10rows_v2: list[str] = ["1,42.0", "2,17.5", "3,88.0", "4,9.5"]11
12print(len(rows_v1))  # -> 313print(len(rows_v2))  # -> 414print(short_hash(rows_v1))  # -> 04cb15print(short_hash(rows_v2))  # -> 8c4016print(short_hash(rows_v1) == short_hash(rows_v1[::-1]))  # -> True17print(short_hash(rows_v1) == short_hash(rows_v2))  # -> False

Three rows against four, order flipped still matches, content changed never matches. The prints 04cb and 8c40 are the real outputs for these rows. Log the 4-char pointer beside every metric and same-data questions become string compares. Swap the slice to 12 chars for real use so collisions stay out of reach.

Watch Out For

The Quick Version

• A model is code plus rows, so pinning code alone reproduces nothing when tables move.
• Hash bytes in sorted leaf order so the pointer depends on content, not on export order.
• Three rows point at 04cb while four point at 8c40, with one added row making a whole new object.
• Store bytes under the hash remotely and never overwrite, so old commits keep reading old rows.
• Commit the small pointer with code so checkout restores the exact rows that trained the model.
• Lineage names the sources and jobs, so a bad upstream table maps to the models that need retraining.

What to Read Next

• Experiment Tracking
• Reproducibility in MLOps: Same Model Twice
• ML Metadata