Data Drift and Distribution Shift

Why Does This Exist?

Six months after launch, a model that was worth money is not. Monitoring shows eleven features drifting, and nobody can say which of them matters.

The word is doing too much work. Three genuinely different things get called drift, they need three different responses, and one of them is invisible to the monitoring most teams have. This page names the three, shows how to detect each, and says what fixes each. The serving skew that looks like drift but comes from your own pipeline is in online offline skew.

The one idea you need first

Write $X$ for the inputs and $Y$ for the thing you predict. $P(X)$ is how often each kind of input shows up. $P(Y \mid X)$ is the answer given an input. Every claim below is a claim about which of those two moved. If that notation is new, it is worth ten minutes before continuing, because the three drift types are just the three combinations of those two moving or holding.

Think of It Like This

How It Actually Works

The three, stated precisely

Covariate shift. $P(X)$ moves, $P(Y \mid X)$ holds. A new marketing channel brings younger customers. The relationship you learned is still true, so the model is still right where it has data, and may be extrapolating where it does not. Often needs nothing at all.

Label shift, also called prior shift. The class balance moves while $P(X \mid Y)$ holds. Fraud rates rise in a recession: fraudsters look the same, there are just more of them. Fixed by recalibrating or moving the decision threshold, not by retraining features.

Concept drift. $P(Y \mid X)$ itself changes. The same input now has a different correct answer, because spammers adapted and what used to be a spam signal is now what everyone writes. No amount of reweighting helps, because what you learned is no longer true.

That third one is the expensive case, and no amount of input monitoring detects it. The inputs can be perfectly stable while the answer changes underneath them.

The shapes drift takes

Sudden: a competitor exits, a pricing change ships, a data source is swapped. Gradual: behaviour moves over months. Incremental: small steps, each below your alert threshold, adding up. Recurring: seasonality, which causes an annual false alarm because December genuinely looks nothing like November and it is not drift.

Detecting input drift, without labels

This part is cheap, and it is what most monitoring does.

Per feature, compare a production window against a reference window, where the reference should be your training data and not last week. Usable tests: a population stability index over fixed bins, a Kolmogorov-Smirnov test for continuous columns, a chi-squared test on categorical bins, or a divergence measure like KL divergence.

Better than any of them, and underused: train a domain classifier to tell a training row from a production row. If it cannot, with AUC near 0.5, nothing has moved. If it hits 0.85, something has, and its feature importances tell you exactly which columns. One model instead of two hundred tests.

Concept drift needs labels, and labels are late

There is no way around this. To know whether $P(Y \mid X)$ changed you need to know $Y$, and $Y$ arrives when it arrives: a churn label takes 90 days, a loan default takes years, a support resolution label takes a week. By the time you can measure the thing that actually matters, a quarter has gone.

So monitor proxies in the meantime. The prediction distribution (if the output histogram moves and the inputs did not, something is off), the confidence distribution (mass drifting toward the middle means the model is less sure), and the unseen-category rate, which directly signals that the input space is expanding.

Fixing each one

Retrain on recent data. Weight recent data more heavily. Importance-weight training rows to match the production input distribution, which is the principled correction for covariate shift. Recalibrate or move the threshold for label shift. For concept drift, detect and adapt: a rolling retrain, or an explicit change-point detector feeding a retrain trigger.

The monitoring discipline nobody follows

Set the threshold in advance, per feature, against your training reference. Not watch the dashboard.

And correct for multiple comparisons, or your alerting is worthless. Two hundred features tested daily at $p < 0.05$ gives you ten false alarms a day by construction. See multiple comparisons. Within a fortnight nobody reads the channel. A drift alerting system that cried wolf is worse than none, because it consumed the attention budget that a real alert needed.

Worked example

One feature, binned four ways. In training the bins hold 40, 30, 20 and 10 percent of the rows. This week they hold 25 percent each.

The population stability index sums $(\text{actual} - \text{expected}) \times \ln(\text{actual}/\text{expected})$ over the bins:

Here $e_i$ is the training share in bin $i$ and $a_i$ is this week share. Bin by bin that is 0.0705, 0.0091, 0.0112 and 0.1374, summing to 0.228. Against the usual reading, under 0.1 quiet, 0.1 to 0.25 worth investigating, over 0.25 a real shift, that is a look at it rather than an emergency.

Notice where it came from. The fourth bin went from 10 to 25 percent and contributed 0.137, which is 60 percent of the total from one bin. PSI is dominated by proportional change in small bins, which is usually what you want and occasionally means a rare category tripling drowns out a genuine move in your main bin.

Code

1import numpy as np2
3reference: np.ndarray = np.array([0.40, 0.30, 0.20, 0.10])  # training window, four bins4current: np.ndarray = np.array([0.25, 0.25, 0.25, 0.25])  # this week, same bin edges5shifted: np.ndarray = np.array([0.20, 0.20, 0.25, 0.35])  # a bigger move into the tail6
7def psi(expected: np.ndarray, actual: np.ndarray) -> float:8    """Population stability index. Under 0.1 quiet, over 0.25 a real shift."""9    return float(((actual - expected) * np.log(actual / expected)).sum())10
11print(np.round(current - reference, 2).tolist())  # -> [-0.15, -0.05, 0.05, 0.15]12print(round(psi(reference, current), 4))  # -> 0.2282  worth investigating13print(round(psi(reference, shifted), 4))  # -> 0.5035  a real shift14print(round(psi(reference, reference), 4))  # -> 0.0  nothing moved

The last line is the sanity check worth keeping in a test: a reference compared with itself must score exactly zero.

Watch Out For

The Quick Version

• Three problems share one word. Covariate shift moves $P(X)$, label shift moves the class balance, concept drift moves $P(Y \mid X)$.
• Covariate shift often needs nothing. Label shift needs recalibration or a new threshold. Concept drift needs a new model.
• Sudden, gradual, incremental, recurring. Seasonality mislabelled as drift is an annual false alarm.
• Input drift is detectable without labels: PSI, KS, chi-squared, or a divergence, against your training reference.
• A domain classifier is the best single instrument. AUC near 0.5 means nothing moved. Its importances name the columns.
• Concept drift needs labels, and labels are late. Monitor prediction distribution, confidence and unseen-category rate meanwhile.
• PSI over four bins going from 40/30/20/10 to 25 each is 0.228, and 60 percent of it came from the smallest bin.
• Set thresholds in advance, per feature, and correct for the number of tests. Two hundred tests at $p < 0.05$ means ten false alarms a night.
• An alerting channel people have learned to ignore is worse than no channel.

What to Read Next

• Online / Offline Skew
• Train, Validation and Test Splits
• Data Leakage