Experiment Tracking

Why Does This Exist?

A week of tuning makes fifty runs. Run 2 used learning rate 0.01 with batch 64 and scored 0.93. By run 30 nobody remembers that, because settings lived in scrollback and weights live in a file named final v3 real final. The best model sits on disk and cannot be found.

Training depends on hundreds of choices: learning rate, batch size, seed, code version, data version, environment. Filenames and memory cannot hold fifty combinations of those. This page covers the logging discipline that makes search honest: one row per run, same columns, ranked on shared lineage. The exact rows behind each score live in data versioning, promotion of the winner lives in model registry, and catching bad inputs before they score lives in data quality monitoring.

Basics first: params are inputs chosen before training. Metrics are outputs measured after it, like accuracy and loss. Artifacts are files a run leaves behind, like weights and plots. Lineage is the code hash plus data hash that make a row rerunnable.

Think of It Like This

How It Actually Works

Every run logs the same four groups before anyone asks which run won. Comparison filters to shared lineage first, then sorts.

Params, metrics, artifacts, lineage

Params are learning rate, batch size, architecture name, seed, and regularizer weights. Metrics are validation loss and accuracy logged per epoch so curves survive, not just finals. Artifacts are weights, plots, and sample predictions. Lineage is the git hash, dataset hash, and environment. Without lineage a score is a rumor because code or data may have moved.

Rank on shared lineage only

The table sorts by one metric after filtering to one code hash and one data hash. That filter is the discipline. Monday data against Friday code cannot crown a winner, however far apart the scores sit.

Worked example: three runs, one winner

Same code at 4b29, same data at 9d01, three configs. Run 7f3a at learning rate 0.02 with batch 32 scores 0.84. Run 2b9e at 0.01 with batch 32 scores 0.89. Run 9c1d at 0.01 with batch 64 scores 0.93.

Rank them:

Winner beats worst by:

Nine points on shared lineage, large enough to ship. Pairs isolate one change each. Learning rate effect:

Batch size effect:

That is the diagram above: three bars at 0.84, 0.89, and 0.93 with the 0.09 gap marked. The rate change bought five points, the batch change bought four, and both claims rest on rows whose lineage matches.

What it costs

One log call per run, kilobytes of metadata, megabytes per kept model. The real price is discipline: every launch path must log, including the midnight manual script. Skip that one and the winner is exactly the run that cannot be explained.

Code

The point is the ranking: rebuild the worked example with stdlib only and confirm the order plus the 0.09 gap.

1runs: list[dict] = []2
3
4def log_run(run_id: str, git: str, data: str, lr: float, batch: int, acc: float) -> None:5    runs.append({"run": run_id, "git": git, "data": data, "lr": lr, "batch": batch, "acc": acc})6
7
8log_run("run_7f3a", "4b29", "9d01", 0.02, 32, 0.84)9log_run("run_2b9e", "4b29", "9d01", 0.01, 32, 0.89)10log_run("run_9c1d", "4b29", "9d01", 0.01, 64, 0.93)11
12same = [r for r in runs if r["git"] == "4b29" and r["data"] == "9d01"]13ranked = sorted(same, key=lambda r: float(r["acc"]), reverse=True)14
15print([r["run"] for r in ranked])  # -> ['run_9c1d', 'run_2b9e', 'run_7f3a']16print(f"{float(ranked[0]['acc']) - float(ranked[2]['acc']):.2f}")  # -> 0.0917print(f"{float(ranked[1]['acc']) - float(ranked[2]['acc']):.2f}")  # -> 0.05

Order matches the hand rank, gaps match 0.93 minus 0.84 and 0.89 minus 0.84, and the winner carries lineage that makes it rerunnable. Remove the lineage filter and the sort still prints, but the winner means nothing.

Watch Out For

The Quick Version

• One tracked run carries params, per-epoch metrics, artifacts, and lineage with code and data hashes.
• Three runs on shared lineage score 0.84, 0.89, and 0.93, so run 9c1d wins by 0.93 minus 0.84, which is 0.09.
• Pairs that differ in one setting isolate its effect: learning rate bought 0.05, batch size bought 0.04.
• Rows on different code or data versions are not comparable, however far apart scores sit.
• Log every launch path including manual midnight scripts, because the untracked run is always the winner.

What to Read Next

• Data Versioning
• Model Registry Platform: Stages to Rollback
• Data Quality Monitoring