Skip to content
AI360XpertBeta

Model Versioning

A model version pins code, data, and environment together. Rebuild any one of the three differently and you get a different model.

Version 2.1 pins git a3f89b, DVC 8f2b9c, and Docker 9d4e to score 0.81, while swapping data to 7c1a02 drops the same code to 0.63
Version 2.1 pins git a3f89b, DVC 8f2b9c, and Docker 9d4e to score 0.81, while swapping data to 7c1a02 drops the same code to 0.63

Why Does This Exist?

A fraud model starts flagging every checkout as fraud on a Tuesday morning. The code has not changed in six weeks, so git log shows nothing useful. The break came from elsewhere: last night retraining read a 50 GB transaction dump where a vendor renamed one column, and the serving host moved from scikit-learn 1.2 to 1.3 with a different default. Same script, different world, different model.

Two ideas carry the rest. A Git hash names the exact code, meaning the commit the trainer ran. A DVC hash names the exact data, meaning the byte content of the training dump stored outside Git. This page covers the rule that binds those two plus the environment into one immutable version. Promoting that version through staging to production lives in model registry. Rerunning it bit for bit lives in reproducibility. Catching it misbehaving live lives in model monitoring.

Think of It Like This

The soup box with recipe, ingredients, and oven

A restaurant ships its signature soup as one sealed box. Inside is the written recipe, the exact bag of tomatoes picked that morning, and a photo of the oven dial at 180 degrees. Cook from that box and you get the same soup.

Swap the tomato bag for a cheaper one while keeping the recipe card, and the soup changes. Keep the tomatoes but bake in a hotter oven, and it changes again. The recipe alone never determined the bowl.

Where it stops: soup ingredients get used up, while data and Docker images get reused without being consumed. The box metaphor also hides storage cost, since a 50 GB dump lives in object storage with only its hash in Git, not in the box itself.

How It Actually Works

Call CC the code hash, meaning the Git commit. Call DD the data hash, meaning the DVC content hash of the training dump. Call EE the environment hash, meaning the Docker image digest plus the lockfile. A model version is the triple:

V=(C,D,E)V = (C, D, E)

Training is a function M(V)M(V), meaning the weights you get from that exact triple. Change any entry and you have a different VV, so you must expect a different MM. Pinning all three is what makes a rerun land on the same weights.

Worked example: same code, one hash swapped

Version 2.1 pins CC to a3f89b, DD to 8f2b9c, and EE to sha256:9d4e. Its held out fraud score is 0.81. Version 2.2 keeps CC and EE identical and swaps DD to 7c1a02, the dump with the renamed column. Its score is 0.63.

0.81−0.63=0.180.81 - 0.63 = 0.18

The 0.18 gap came from data alone, which is why the diagram draws two triads side by side with only the middle hash changed. The data habit behind the DD hash lives in data versioning.

Code

The point is the check: two triads that differ in one hash must never share a version id, and the score gap is 0.18.

v21: dict[str, str] = {"git": "a3f89b", "dvc": "8f2b9c", "docker": "sha256:9d4e"}v22: dict[str, str] = {"git": "a3f89b", "dvc": "7c1a02", "docker": "sha256:9d4e"}
score_v21: float = 0.81score_v22: float = 0.63
print(f"{score_v21:.2f} {score_v22:.2f} {score_v21 - score_v22:.2f}")  # -> 0.81 0.63 0.18print(v21 == v22)  # -> False

The printout confirms the two versions differ as dicts and the retrain drops 0.18. Change v22 back to 8f2b9c and the equality flips to True.

Watch Out For

Versioning by date instead of by hash

Symptom: the run sheet says trained on the users table as of Tuesday 5 PM, yet checking out Tuesday code and pulling Tuesday data gives score 0.77 instead of the logged 0.81. A Wednesday UPDATE rewrote Monday rows in place, so Tuesday no longer exists anywhere.

Fix: never trust a timestamp on a mutable table. Store the DVC content hash or a time travel snapshot id from Delta Lake in the version triple, and refuse to promote a model whose data hash cannot be fetched.

Floating the environment tag

Symptom: staging passes at 0.81 and production scores 0.74 on the same version id. The version pinned python:3.11 instead of a digest, and the base image moved sklearn 1.2 to 1.3 between the two deploys.

Fix: pin EE to the image digest such as sha256:9d4e plus the lockfile hash, never to a moving tag. Repoint the tag only by cutting a new version, then requalify through the model registry flow.

The Quick Version

  • A model version is the triple of code hash, data hash, and environment digest.
  • Version 2.1 pins git a3f89b, DVC 8f2b9c, and Docker sha256:9d4e for fraud score 0.81.
  • Swapping only the data hash to 7c1a02 gives version 2.2 at 0.63, a gap of 0.18.
  • Git alone cannot reproduce a model because data and environment move independently.
  • Large dumps stay in object storage while their hashes live in Git beside the code.
  • Promotion, reruns, and live alarms belong to registry, reproducibility, and monitoring pages.