Core ML

CI/CD for Machine Learning

In software, merging code to 'main' triggers a server deployment. In ML, merging code to 'main' should trigger a training pipeline, not a deployment. The model itself requires a second, separate CI/CD loop.

Standard CI/CD has two loops: Code -> Test -> Deploy. ML introduces a third loop (Continuous Training): Code -> Test Pipeline -> Train Model -> Test Model -> Deploy.

Why Does This Exist?

In traditional Software Engineering (DevOps), CI/CD (Continuous Integration / Continuous Deployment) is a solved problem.

A developer writes code.
They open a Pull Request.
CI runs unit tests.
They merge to main.
CD automatically deploys the code to the production server.

If you apply this exact pipeline to Machine Learning, your company will burn down. If a Data Scientist changes the learning rate from 0.01 to 0.05 and merges to main, you cannot automatically deploy that to the production server. You don't even have a model yet! You just have a script that might produce a good model in 14 hours.

CI/CD for ML (MLOps) requires splitting the pipeline into two distinct halves: testing the Code, and testing the Model.

Think of It Like This

Imagine a car factory. Traditional CI/CD is like upgrading the robots on the assembly line. If the upgrade passes the safety tests, you turn the robots on. But turning on the robots does not immediately put a new car in a customer's driveway. The robots first have to build the car. Then the car itself has to be crash-tested. Only if the car passes the crash test does it get shipped.

In ML, your Python code is the robot. The Model is the car.

How It Actually Works

A mature MLOps pipeline introduces a new concept called Continuous Training (CT). The workflow looks like this:

1. CI (Continuous Integration): Testing the Pipeline

A data scientist opens a Pull Request changing the feature engineering logic. GitHub Actions runs standard CI. It does not train the massive model. It just runs pytest to ensure the data transformation functions don't crash, and it runs a tiny, dummy training loop (e.g., 1 epoch on 100 rows) just to ensure the code compiles. If CI passes, the code is merged to main.

2. CT (Continuous Training): Building the Model

Merging to main triggers the CT phase. The orchestrator (like Airflow or Kubeflow) spins up the massive GPU cluster, pulls the latest data, and trains the model for 14 hours. It logs everything to the Experiment Tracker. When training is done, it produces a candidate model.pkl.

3. Model CI: Testing the Model

We now have a model artifact. Before we deploy it, we run Model CI. We run the model against the Golden Set (as discussed in testing-ml-systems). We assert that F1_score > 0.85 and that it passes all Behavioral Invariance tests. If it passes, the model is pushed to the Model Registry in the Staging phase.

4. CD (Continuous Deployment): Shipping the Model

Finally, the deployment phase. Depending on company policy, this might be fully automated or require a manual click. CD pulls the Staging model from the registry, wraps it in a FastAPI server, containerizes it via Docker, and rolls it out to the Kubernetes cluster, officially transitioning it to Production.

Show Me the Code

Here is a conceptual GitHub Actions .yml file showing the difference between Code CI and Model CI.

1name: ML CI/CD Pipeline2
3on:4  push:5    branches: [ main ]6
7jobs:8  # 1. CODE CI (Runs in 2 minutes)9  test_pipeline_code:10    runs-on: ubuntu-latest11    steps:12      - uses: actions/checkout@v313      - name: Run Pytest on Data Transformations14        run: pytest tests/test_features.py15      - name: Dry Run Training (1 Batch)16        run: python train.py --epochs 1 --dummy_data True17
18  # 2. CONTINUOUS TRAINING (Runs in 14 hours, triggered by CI passing)19  trigger_kubeflow_training:20    needs: test_pipeline_code21    runs-on: ubuntu-latest22    steps:23      - name: Trigger Remote Training DAG24        run: curl -X POST https://airflow.mycompany.com/api/v1/dags/train_model/dagRuns25
26  # ... The training finishes 14 hours later and triggers a webhook ...27
28  # 3. MODEL CI (Runs in 10 minutes, testing the actual weights)29  test_model_artifact:30    runs-on: ubuntu-latest31    steps:32      - name: Download Model from Registry33        run: mlflow artifacts download -u models:/churn_model/Staging34      - name: Run Golden Set Evaluation35        run: pytest tests/test_model_behavior.py36
37  # 4. CD38  deploy_to_production:39    needs: test_model_artifact40    steps:41      - name: Deploy to Kubernetes42        run: kubectl apply -f deployment.yaml

Watch Out For

Skipping Continuous Training (CT) on Data Drift. Most teams build pipelines that only trigger when someone pushes code. But in ML, the code is only half the equation. If the underlying user behavior changes (Data Drift), your model's accuracy will quietly degrade in production even though no one touched main. A true MLOps pipeline also triggers the CT phase on a schedule (e.g., every Sunday night) or on an alert (e.g., when the monitoring system detects drift), retraining the model without any human intervention.

The Quick Version

Standard CI/CD doesn't work for ML because merging code doesn't produce an app; it produces a script that might train a model.
MLOps introduces Continuous Training (CT).
Code CI tests if the Python scripts compile and transform data correctly.
CT trains the actual model on the cloud using the new code.
Model CI tests the resulting model weights against a Golden Set to ensure it is accurate.
CD finally wraps the verified model in an API and deploys it to production.

Related concepts

Model Registry