Model Drift Detection

Why Does This Exist?

A recommendation model trained on summer buying keeps serving in December. No exception fires, latency stays flat, and revenue slides for weeks before anyone asks whether the model still matches the world. The model did not break. The world moved: incomes shifted with inflation, tastes changed, a sensor aged. A frozen function of a moving world decays by default.

Drift detection is the scheduled comparison that catches that move. A job pulls recent inference rows, compares their distributions and the prediction mix against the training baseline, and pages when the distance passes a threshold. It answers one question nightly: should we retrain today. Without it, the first detector is a customer complaint.

This page covers the comparison itself: data drift, concept drift, and the alerting loop. Testing gates before deploy are ML testing strategies. Watching nulls and schema live is data quality monitoring. Owning the whole serving picture is model monitoring. When an alert fires, ML metadata names the exact training snapshot to compare against.

Basics first: call the input distribution $P(X)$, which means how often each feature value appears. Call the relationship $P(Y|X)$, which means the right answer given those features. Drift moves one or both, and the two moves need different responses.

Think of It Like This

How It Actually Works

Two drift kinds share one pipeline. Data drift moves $P(X)$ while the relationship holds: average income rises, but income still predicts risk the same way. Concept drift moves $P(Y|X)$ itself: the same income now means different risk because behavior changed. Prediction drift watches the output mix $P(\hat{Y})$ as a proxy when true labels arrive weeks late.

Distance with PSI over shared bins

Bin one feature the same way in training and live traffic, then compare shares. Call expected share $E_i$, which means the training fraction in bin $i$. Call actual share $A_i$, which means the live fraction in the same bin. The Population Stability Index is $PSI = \sum (A_i - E_i) \times \ln(A_i / E_i)$, summed over bins. Teams commonly treat below 0.10 as noise, 0.10 to 0.25 as review, and above 0.25 as a major shift. Thresholds differ by team, but the arithmetic is shared.

The nightly loop with four steps

Pull the last window of inference rows with inputs and predictions. Compute distances against the saved baseline on guarded features plus the prediction mix. Compare each score to its threshold, with seasonality handled by comparing against the same season last year or a rolling window. On breach, page with the top drifting feature and route to accuracy checks, then to retraining when harm is confirmed.

Worked example

One income feature uses three bins. Training shares over 1,000 rows are 0.30, 0.50, and 0.20, which means counts of 300, 500, and 200 since $0.30 \times 1,000 = 300$. Live shares over 500 rows are 0.20, 0.45, and 0.35, which means counts of 100, 225, and 175.

Per-bin contributions are $(0.20 - 0.30) \times \ln(0.20 / 0.30) = 0.0405$, plus $(0.45 - 0.50) \times \ln(0.45 / 0.50) = 0.0053$, plus $(0.35 - 0.20) \times \ln(0.35 / 0.20) = 0.0839$. The sum is $0.0405 + 0.0053 + 0.0839 = 0.1298$, reported as PSI 0.13. The team review line sits at 0.10, and 0.13 clears it, so the job pages with the third bin named as the driver.

Code

The point is the distance decision with stdlib only: same shares as the worked example, same 0.13, same threshold check.

1import math2
3expected = [0.30, 0.50, 0.20]4actual = [0.20, 0.45, 0.35]5
6parts = [(a - e) * math.log(a / e) for a, e in zip(actual, expected)]7psi = sum(parts)8print([round(p, 4) for p in parts])  # -> [0.0405, 0.0053, 0.0839]9print(round(psi, 4))  # -> 0.129810print(round(psi, 2) > 0.10)  # -> True11
12train_counts = [int(e * 1000) for e in expected]13live_counts = [int(a * 500) for a in actual]14print(train_counts, live_counts)  # -> [300, 500, 200] [100, 225, 175]

Three contributions, one sum at 0.1298, and a boolean that pages. That boolean is the whole nightly job: distance, threshold, alert.

Watch Out For

The Quick Version

• Drift detection compares live traffic to the training baseline on a schedule and pages past a threshold.
• Data drift moves inputs $P(X)$, concept drift moves the relationship $P(Y|X)$, and prediction drift proxies the second.
• PSI sums $(A_i - E_i) \times \ln(A_i / E_i)$ over shared bins, with 0.10 as a common review line.
• Shares of 0.30, 0.50, 0.20 against 0.20, 0.45, 0.35 give 0.0405 plus 0.0053 plus 0.0839 for PSI 0.13.
• Seasonality needs same-season or rolling baselines, or every summer pages falsely.
• Distance without accuracy harm wastes retrains, so confirm before rebuilding.
• Fired alerts resolve through lineage, which names the exact training snapshot to beat.

What to Read Next

• Model Monitoring Platform: Drift to Alert
• Data Quality Monitoring
• ML Metadata