CI/CD in ML Pipelines
ML automation tests data and retrains models alongside code. New rows, drift alerts, or a schedule can all trigger a fresh model, not just code commits.
Why Does This Exist?
Classic CI/CD assumes code is the only thing that changes. That assumption quietly rots ML systems. A model left untouched still degrades as language, prices, fraud patterns, and user habits move underneath it. Teams that deploy on code commits alone end up serving stale models with green build badges.
ML pipelines add a third loop, continuous training, next to integration and deployment. Retraining triggers on data drift and schedules as well as commits, and every candidate must beat the live baseline before it ships. This page covers the three loops and their triggers. Versioning the data they consume is data versioning, recording what each run tried is experiment tracking, and testing the model itself is ML testing strategies. Approved images ship sealed through containerization and roll out gated through canary deployments.
Basics first: concept drift means the relationship between inputs and targets changed, so yesterday's model is wrong about today even with perfect code. A drift monitor watches live predictions and data statistics, then fires a retraining trigger when they move past a threshold. The pipeline below is what that trigger starts.
Think of It Like This
A bakery that tastes the flour every morning
A factory stamps identical parts once its machines pass inspection. A bakery cannot work that way: today's flour shipment differs from yesterday's, so the same recipe and oven still produce different bread.
This bakery tastes and weighs the flour each morning, adjusts water and bake time, and only serves loaves that pass the taste panel. Recipe checks are integration, adjusting to the flour is training, and the taste panel before the shelf is deployment gating.
Where it stops: a bakery adjusts within one recipe. When customers want sourdough instead of white, no amount of tweaking saves the old formula. Pipelines retrain models; rethinking the product is still your job.
How It Actually Works
Three loops hand off to each other, each with its own trigger and gate.
- Continuous integration tests code. A commit carrying new features or architecture runs unit tests, linting, and a tiny smoke train on a data sample. The output is a build of the training pipeline, not a model. This loop is fast by design.
- Continuous training tests data and trains. Triggered by a schedule, a drift alert, or fresh data landing, this loop validates the dataset schema, checks null and range statistics, trains on the full data, and evaluates on a frozen holdout. A candidate that fails data checks or trails the baseline never reaches the registry.
- Continuous deployment ships gated models. An approved registry version packages into a prediction service and rolls out behind a gate, often a canary. Live metrics close the loop back to the drift monitor.
Worked example
Overnight, 40,000 fresh rows land and the schedule fires the training loop. Validation finds the new batch clean, so training runs and the candidate scores AUC 0.83 on the frozen 5,000-row holdout against the live baseline at 0.81: an uplift of 0.02, about 2.5 percent in relative terms. That clears the promotion bar, so the registry accepts v7 and deployment rolls it out gated. Had validation instead found a quarter of rows missing a key field, as in the snippet below, the run would stop before spending a cent on training.
Code
Validation runs before training, always. This check enforces the schema and a null-rate ceiling on an incoming batch, using only the standard library.
def validate_batch(rows: list[dict], ceiling: float = 0.05) -> str: """Block retraining when the schema breaks or nulls exceed the ceiling.""" expected = {"age", "fare"} nulls = sum(1 for r in rows if any(r.get(k) is None for k in expected)) rate = nulls / len(rows) print(f"null rows: {nulls}/{len(rows)} = {rate:.1%}") # -> null rows: 1/4 = 25.0% return "block retrain" if rate > ceiling else "pass"
rows = [ {"age": 34, "fare": 12.5}, {"age": None, "fare": 8.0}, {"age": 29, "fare": 41.2}, {"age": 51, "fare": 7.9},]print(validate_batch(rows))# -> block retrainprint(f"auc new=0.83 base=0.81 uplift={(0.83 - 0.81):.2f}")# -> auc new=0.83 base=0.81 uplift=0.02One null in four rows is a 25 percent null rate against a 5 percent ceiling, so the batch blocks. That single gate saves a full training run on broken data.
Watch Out For
Retraining on every commit
Wiring full training to every merge burns money for nothing. Most commits touch docs, dashboards, or serving code that changes no model, yet each one would launch hours of GPU time. Worse, commit-triggered models churn the registry with near-identical versions nobody can tell apart.
Trigger cheap checks on commits and full training on data or drift. Commits run unit tests plus the smoke train on a sample. Nightly arrivals, drift alerts, and explicit requests run the full loop. Your GPU bill drops and every registry version has a reason to exist.
A silent schema change that trains a crippled model
Upstream renames a column from user_age to age_years and the pipeline keeps running. Feature code reads a missing key, fills a default, and trains a model that never saw real ages. Validation that only checks row counts waves it through, and the gated deployment ships a model that is confidently wrong.
Assert the schema first: exact column names, types, and value ranges, before any training step. The snippet above is the minimum. Fail the run loudly at this gate rather than debugging a mysterious metric drop a week later.
The Quick Version
- ML CI/CD adds continuous training to integration and deployment, because drift rots models that no commit touched.
- Integration tests code fast, training validates data and retrains, and deployment ships only registry-approved versions behind a gate.
- Triggers include schedules, drift alerts, and fresh data, not just commits.
- In the worked run, a candidate at AUC 0.83 beat the 0.81 baseline by 0.02 and shipped, while a batch with 25 percent nulls blocked before training.
- Validate schema and null rates first, and never fire full training on every commit.