MLOps Overview

Notebooks prove an idea, production proves a system. MLOps is the loop of training, validating, deploying, and monitoring that carries a model from one to the other without silent decay.

A model scoring 0.91 in the notebook falls to 0.78 in production for a gap of 0.13, so the monitor fires and the loop retrains instead of decaying quietly

Why Does This Exist?

A notebook model scores 0.91 and everyone celebrates. Three months later the same model scores 0.78 on live traffic and nobody noticed, because no job re-ran training, no check compared versions, and no alert watched predictions. The code did not crash. The endpoint returned success. The business quietly paid for 13 points of decay. That gap between a working notebook and a reliable system is where most ML projects fail.

MLOps closes it by turning a one-off experiment into a repeatable loop: version the data, test the code, train the model, evaluate it, register the winner, deploy it, and watch it. Each step is automated, each artifact is versioned, and monitoring feeds back into retraining. Manual work becomes the exception, not the process.

This page maps the whole loop and its maturity stages. The ledger that links each model to its data is ML metadata. The files that rebuild identical environments are infrastructure as code. Per-run parameter logging is experiment tracking, and automating the loop on every change is CI/CD for ML.

Basics first: training-serving skew means the features a model sees live differ from the ones it trained on, like an average computed with different code in two places. Averages of 42.32 in training and 38.90 in serving do not look broken, but the model experiences a different world. The loop below exists largely to remove that class of surprise.

Think of It Like This

The bakery that became a production line

A home baker improvises: a pinch of this, a hot oven, a good loaf, an unrepeatable recipe. A commercial bakery runs a line: weighed recipes, timed mixes, a check after each stage, freshness dates on every shelf, and a reorder when stock runs low. One bad batch traces to one lot of flour in minutes.

MLOps is the production line for models. Versioned data is the weighed recipe. Pipeline tests are the stage checks. The registry is the shelf with dates. Monitoring is the freshness check that pulls a stale loaf and bakes again.

Where it stops: a line reproduces a recipe faithfully, including a bad one. Process guarantees repeatability and early warnings, not a good model. A weak idea automated at Level 2 fails faster, not better.

How It Actually Works

Seven steps form the loop. Call training accuracy $A_{test}$ , which means score on held-out data before deploy. Call live accuracy $A_{live}$ , which means score on recent production rows once labels arrive. Call the gap $G = A_{test} - A_{live}$ , which means how much the world moved. When $G$ passes a threshold, the loop retrains.

The seven steps in order

Version the dataset snapshot, so retraining can return to the exact rows.
Test pipeline code and validate incoming data before training starts.
Train with logged parameters, code commit, and environment.
Evaluate on held-out rows plus guarded slices and behavior probes.
Register the winner with its lineage, stage, and approval.
Deploy behind a serving path with health and behavior checks.
Monitor predictions and inputs, and trigger retraining on decay.

Skipped steps do not disappear; they become manual work that fails at 2am. The maturity model below counts how many of the seven run on their own.

Maturity counts automation

Level 0 runs 0 of 7 automatically. Notebooks train, scripts copy files, no monitor watches. Fine for learning, fragile for users.

Level 1 automates 4 of 7: versioned training, logged runs, evaluated artifacts, and a registered model. Retraining runs on schedule or new data, but deploy and monitoring stay manual.

Level 2 automates 7 of 7. A data, code, or parameter change triggers tests, training, evaluation, and conditional deploy. Monitoring alerts trigger investigation or retraining without a human starting the job.

Worked example

A fraud model scores 910 correct of 1,000 held-out rows, so $A_{test} = 910 / 1,000 = 0.91$ . One month later 1,000 production rows with arrived labels score 780 correct, so $A_{live} = 780 / 1,000 = 0.78$ . The gap is $G = 0.91 - 0.78 = 0.13$ . The team threshold is 0.05, and 0.13 is above it, so the monitor fires and the loop retrains from the versioned snapshot instead of serving a stale model for another quarter.

Code

The point is the loop decision in numbers: count automated steps for the level, then compare test accuracy to live accuracy with stdlib only.

1steps = ["version_data", "test_code", "train", "evaluate", "register", "deploy", "monitor"]2
3level_1 = ["version_data", "train", "evaluate", "register"]4level_2 = steps5print(len(level_1), len(level_2), len(steps))  # -> 4 7 76
7a_test = 910 / 10008a_live = 780 / 10009gap = a_test - a_live10print(round(a_test, 4), round(a_live, 4), round(gap, 4))  # -> 0.91 0.78 0.1311print(gap > 0.05)  # -> True

Four of seven automated at Level 1, seven of seven at Level 2, and a gap of 0.13 above the 0.05 threshold. That boolean is the retrain decision the monitor makes nightly.

Watch Out For

Lifting DevOps without the data half

The pipeline deploys on every commit, tests pass, latency looks fine, and predictions are stale. DevOps tooling versions code while data, features, and statistical behavior drift unversioned. The symptom is reliable deploys of useless models.

Add the ML-native steps: versioned datasets, training-serving parity checks, slice evaluation, and prediction monitoring. A deploy pipeline without those is DevOps wearing an MLOps badge.

Watching latency while accuracy decays

The endpoint returns success in milliseconds, so infra dashboards stay green while $A_{live}$ slides from 0.91 to 0.78. System health and model health are different signals, and only the second says whether the model still works.

Monitor prediction and feature distributions alongside latency and errors, with one threshold that pages a human, like $G$ above 0.05. Latency tells you the server is alive. Behavior tells you the model is useful.

The Quick Version

MLOps is a loop of version, test, train, evaluate, register, deploy, and monitor that keeps models fresh.
Level 0 automates 0 of 7 steps, Level 1 automates 4 of 7, Level 2 automates 7 of 7.
Training-serving skew is the silent killer the loop is built to remove.
A model at 0.91 in test and 0.78 live has a gap of 0.13, which fires a 0.05 threshold into retraining.
Versioned data plus logged runs plus monitoring turns decay into a scheduled repair.
Infra health without behavior health hides stale models behind green dashboards.