Experiment Tracking
Log the settings, code version, and data version behind every training run, then rank runs by score. The log turns the good model from a memory into a row you can rerun.
Why Does This Exist?
A week of tuning produces fifty runs. Run 2 used learning rate 0.01 with batch size 64 and scored 92 percent. By run 14 nobody remembers that, because the settings lived in a terminal scrollback and the weights live in a file called model final v3 really final. The best model is lost while sitting on disk.
The failure is structural, not forgetfulness. Training depends on hundreds of choices: learning rate, batch size, architecture, seed, code version, data version. A human cannot hold fifty combinations of those in memory, and a filename cannot either.
Two ideas carry the rest. A hyperparameter is any setting chosen before training rather than learned by it, and the search over those settings is hyperparameter tuning. This page covers the logging discipline that makes the search honest: one row per run, same columns every time, ranked by score. Promoting the winning row to production lives in the model registry, and rerunning a row bit for bit lives in reproducibility.
Think of It Like This
A baker's numbered loaf log
A baker tests sourdough for a month. Each loaf gets a number, and the log records flour blend, water temperature, and bake time beside the loaf number and the family's score out of ten. Loaf 23 scores nine. When the cafe orders that bread, the baker does not guess. The log says exactly what loaf 23 was, and the oven repeats it.
The mapping is direct. Each loaf is a training run, the recipe is the hyperparameter configuration, the family score is the validation metric, and the loaf number is the run id that ties them together.
Where it stops: a loaf has three variables measured by hand. A training run has hundreds of settings plus the code and the data underneath them, and both shift daily. A paper log works for bread. Training needs the log to be automatic, with version hashes instead of handwriting.
How It Actually Works
Every run logs the same four groups before anyone asks which run won.
Params are the inputs: learning rate, batch size, architecture name, seed, regularizer weights. Metrics are the outputs: validation loss and accuracy, logged at every epoch so the full curve survives, not just the final number. A curve that bottoms at epoch 12 and climbs after tells you when overfitting started. Artifacts are the files: the saved weights, the confusion plot, sample predictions. Lineage is the context: the git commit hash of the code, the hash of the dataset, and the environment, so the row can be rerun exactly.
The comparison is a table sorted by one metric, filtered to rows that share code and data versions. That filter is the whole discipline. Without it you compare a score from Monday's data against a score from Friday's code, and the winner means nothing.
Worked example: three runs, one winner
Same code at git 4b29, same data at hash 9d01, three configurations:
Rank them: 0.92, then 0.88, then 0.81. The winner beats the worst by:
Eleven points on shared lineage, large enough to act on. The pairs isolate one change each because everything else matches. Runs 3c9d against 1a4b differ only in learning rate:
Runs 1a4b against 8f2a differ only in batch size:
That is the diagram above: three rows, one sorted table, the winner marked by 0.11. The learning rate change bought seven points, the batch change bought four, and both claims rest on rows whose lineage matches.
Why lineage is the mechanism, not decoration
A score without a code version is a rumor. Suppose run 8f2a had trained on Tuesday's data while the others used Monday's. The 0.11 gap could be cleaner labels, not a better learning rate, and shipping 8f2a's settings would ship nothing. The git hash and data hash turn each row from a claim into a rerunnable fact. If two rows disagree on lineage, they are not comparable, however far apart their scores sit.
What it costs
One log call per run, a few kilobytes of metadata, and megabytes of artifacts per kept model. The real price is discipline: every launch path must log, including the quick manual script run at midnight. Skip that one and the winner is exactly the run you cannot explain. There is no case where skipping pays, because an untracked winner is a model you may not ship.
Code
The point is the ranking: rebuild the worked example with plain stdlib Python and confirm the order plus the 0.11 gap.
import json
runs: list[dict] = []
def log_run(run_id: str, git_sha: str, data_hash: str, lr: float, batch: int, acc: float) -> None: runs.append({"run": run_id, "git": git_sha, "data": data_hash, "lr": lr, "batch": batch, "acc": acc})
log_run("run_3c9d", "4b29", "9d01", 0.05, 64, 0.81)log_run("run_1a4b", "4b29", "9d01", 0.01, 32, 0.88)log_run("run_8f2a", "4b29", "9d01", 0.01, 64, 0.92)
ranked = sorted(runs, key=lambda r: float(r["acc"]), reverse=True)print([r["run"] for r in ranked]) # -> ['run_8f2a', 'run_1a4b', 'run_3c9d']print(f"{float(ranked[0]['acc']) - float(ranked[2]['acc']):.2f}") # -> 0.11print(json.dumps(ranked[0]["git"]) + " " + json.dumps(ranked[0]["data"])) # -> "4b29" "9d01"The order matches the hand ranking, the gap matches 0.92 minus 0.81, and the winner carries the lineage that makes it rerunnable. In real tooling the sort is preceded by a lineage filter that keeps only rows sharing one git hash and one data hash. Without that filter the ranking still prints, but the winner means nothing.
Watch Out For
Logging the data path instead of the data hash
The row says data at s3://bucket/train.csv. A week later someone overwrites that file. You rerun the exact settings from the log and get a different model, because the path stayed the same while the bytes changed. Nothing errors. The tracker shows identical inputs with different outputs.
Symptom: reruns of logged configurations drift without any code change, usually right after a data refresh. Fix: log the hash or version id of the dataset with every run, never the bare path, and treat a changed hash as a different experiment even when the settings match.
Declaring a winner across different code versions
Three runs score 0.81, 0.88, and 0.92, and the 0.92 row ran on an older commit with a different preprocessing function. The table still sorts, the dashboard still crowns it, and the team ships settings whose score belonged to code that no longer exists.
Symptom: the winner's git hash differs from the runners up, or the best row cannot be reproduced from the current main branch. Fix: filter every comparison to one git hash and one data hash before sorting, and promote winners to the model registry only from the current lineage.
The Quick Version
- One tracked run carries params, per-epoch metrics, artifacts, and lineage with the git hash and data hash.
- Our three runs on shared lineage score 0.81, 0.88, and 0.92, so run 8f2a wins by 0.92 minus 0.81, which is 0.11.
- Pairs that differ in exactly one setting isolate its effect: learning rate bought 0.07, batch size bought 0.04.
- Rows on different code or data versions are not comparable, however far apart their scores sit.
- Log every launch path including manual midnight scripts, because the untracked run is always the winner.