Model Monitoring Platform: Drift to Alert

Why Does This Exist?

A deployed model fails silently. It keeps returning answers with a 200 status while its predictions get worse, because the world moved and the weights did not. New customers arrive, prices shift, fraud rings change tactics, and the training distribution becomes history.

This page covers the platform that catches that decay: collectors, baselines, detectors, alert routing, and the retraining trigger. The run-level view in Model Monitoring teaches what to watch on one model. Here you learn how a team watches a hundred of them without drowning. When a detector fires, the cause is usually drift, which is why model drift detection is the natural next read, and bad inputs are often a data problem first, so data quality monitoring sits beside it.

The one idea you need first

A baseline is the reference picture of healthy behavior: the feature distribution at training time, the usual output mix, the normal latency band. Every live window gets compared against it. Without a stored baseline there is nothing to drift from, only vibes.

Think of It Like This

How It Actually Works

Three collectors feed one decision. Input features catch drift before it costs money. Output mix catches behavior change the features miss. System stats separate a sick model from a sick cluster. Each live window, usually an hour or a day of traffic, is scored against the baseline, and a rule decides: page, ticket, or silence.

The three signals

Input drift asks whether the features arriving now still look like training. Output shift asks whether the prediction mix moved. System health asks whether latency and error rate stayed inside the service band. You need all three because each one goes blind in a different way: labels lag by weeks, outputs can look stable while inputs rot, and a fast 200 response can still carry a useless score.

Worked example

Take an income feature with three brackets. Training shares are $P = [0.50, 0.30, 0.20]$. This week the live shares are $Q = [0.28, 0.34, 0.38]$. Population Stability Index is the sum over brackets of $(Q_i - P_i) \ln(Q_i / P_i)$, where $P_i$ is the training share and $Q_i$ is the live share:

• Bracket one: $(0.28 - 0.50) \times \ln(0.28 / 0.50) = (-0.22) \times (-0.58) \approx 0.128$
• Bracket two: $(0.34 - 0.30) \times \ln(0.34 / 0.30) = 0.04 \times 0.125 \approx 0.005$
• Bracket three: $(0.38 - 0.20) \times \ln(0.38 / 0.20) = 0.18 \times 0.642 \approx 0.116$

Add them: $0.128 + 0.005 + 0.116 \approx 0.25$. The platform rule says below 0.1 is noise, 0.1 to 0.2 is watch, above 0.2 is investigate. At 0.25 this window pages the owner with the bracket breakdown attached.

The other two signals stay green in the same window, which is itself information. Approval mix moved from 5.0 percent at training to 5.4 percent live, a relative move of 8 percent, inside the 30 percent relative rule. P99 latency sits at 180 ms against a 350 ms objective. So the ticket says drift, not outage, and routes to the data owner instead of on-call infrastructure.

From signal to action

A firing rule does three things: it opens a ticket with the window, the metric, and the baseline attached; it routes to exactly one owner; and it proposes the next step. Small sustained drift schedules a retraining job through CI/CD pipelines. That retrain only counts if it replays, so the job must meet the four pins in reproducibility. A sharp output break holds the release and rolls traffic back. The loop closes only when the new model version becomes the baseline, otherwise the same alert fires again next week and everyone learns to ignore it.

Code

The point of this snippet is the detector itself: a small PSI function you can run on any two distributions, with the same thresholds the platform uses.

1import math2
3
4def compute_psi(expected: list[float], actual: list[float]) -> float:5    """Population Stability Index between baseline and live shares."""6    return sum((a - e) * math.log(a / e) for e, a in zip(expected, actual))7
8
9def disposition(psi: float) -> str:10    """Route the window: silence, watch, or page."""11    if psi > 0.2:12        return "page owner"13    if psi > 0.1:14        return "watch"15    return "silence"16
17
18p = [0.50, 0.30, 0.20]  # training shares19q = [0.28, 0.34, 0.38]  # live shares20psi = compute_psi(p, q)21print(round(psi, 3))  # -> 0.24822print(disposition(psi))  # -> page owner

The output 0.248 rounds to the 0.25 in the worked example, and the disposition matches the ticket the platform opens.

Watch Out For

The Quick Version

• A monitoring platform compares each live window against stored baselines and routes breaches to one owner.
• Watch three signals together: input drift, output mix, and system health, because each goes blind differently.
• PSI above 0.2 on a key feature means investigate; 0.1 to 0.2 means watch; below 0.1 is noise.
• When labels lag by weeks, proxy metrics like approval rate with relative thresholds carry the watch.
• Every alert must propose the next step, retrain or rollback, or the team will learn to ignore it.

What to Read Next

• Model Drift Detection
• Data Quality Monitoring
• CI/CD in ML Pipelines