Data Drift Detection
Today's inputs no longer look like training data, yet the model keeps answering with full confidence. Data drift detection compares the two windows bin by bin and raises the alarm before accuracy slides.
Why Does This Exist?
A model keeps answering after the world moves. It doesn't crash or complain. It just applies last year's pattern to this year's crowd, and the damage shows up weeks before the labels that would prove it.
That silence is the whole problem. Fresh labels rarely arrive in time, but fresh inputs arrive every hour. So you compare today's feature mix against the training mix, one feature at a time, and you'll set the alarm before accuracy slides.
This page covers that comparison for input drift: the PSI sum, the KS check, and one checkout example you can verify by hand. When the right answer itself moves while the inputs look steady, that is concept drift. The wider split of all three shifts lives at data drift and distribution shift, and the loop that acts on the alarm is model monitoring.
One line of grounding before the math. Call training time with distribution , and production with . Data drift is the claim that no longer matches , where means the feature values you feed the model.
Think of It Like This
The coin jar on the counter
You keep a reference jar of 200 coins by the till: 100 small, 60 medium, 40 large. Every evening you scoop that day's 200 coins and sort them into the same three slots. One week the scoop reads 56, 64, 80. Same 200 coins, visibly different mix, and you know the till drifted before the books say so.
The jar is the training window, the evening scoop is the production window, and the three slots are the fixed bins. The PSI score is just the weighted surprise across those slots, with the slot that doubled shouting the loudest.
Where it stops: coins sit still while you count them, and you can see every coin. Model inputs are invisible at scale, which is why the detector is your only eyes. And coins never change what they are worth, so this jar says nothing about the day prices themselves move.
How It Actually Works
Fix the bins from training data and never move them. Write for the expected share of bin at training, meaning the fraction of training rows that fell there, and for the actual share of the same bin in production. Both lists sum to 1. The population stability index adds up each bin's surprise:
Here is the natural log. A bin that keeps its share contributes zero. A bin that doubles contributes a lot, because the log of the ratio grows while the gap in front of it grows too.
The usual reading is under 0.1 quiet, 0.1 to 0.25 worth a look, and over 0.25 a real shift that starts the retrain conversation. Always compare against training data, not last week, or slow drift becomes the new normal without ever tripping the wire.
Worked example: three bins score PSI 0.27
One checkout feature, amount per order, cut into three fixed bins: low, mid, high. The training window holds 200 orders: 100 low, 60 mid, 40 high. So . This week's 200 orders read 56 low, 64 mid, 80 high. So .
Bin by bin, with the log to four places:
- Low:
- Mid:
- High:
Add them:
That clears the 0.25 line, so it is a real shift, not noise. Notice the shape of it. The high bin doubled from 0.20 to 0.40 and supplied 0.1386, over half the total. The low bin nearly halved and supplied most of the rest. The mid bin barely moved and contributes almost nothing. PSI is dominated by proportional moves in a single bin, which is usually what you want to catch.
The KS check on the same bins
The Kolmogorov-Smirnov check asks a simpler question: what is the largest gap between the two cumulative curves, where each curve just adds the shares up to that bin. Write for the training cumulative and for production:
- Training cumulative:
- Production cumulative:
- Gaps: , so
runs from 0 (identical curves) to 1 (no overlap). Here 0.22 on 200 rows per side is a wide gap, and it points at the same low bin as the PSI sum. The two tests agree, which is the outcome you trust. When they disagree, believe the effect size over any p-value, because a p-value shrinks with sample size and will eventually flag a move too small to matter. If this shift holds, the fix is domain adaptation or a retrain, not a wider threshold.
Code
The point is the check: rebuild the two windows from counts and confirm the hand computation of 0.27 and 0.22.
import numpy as np
train: np.ndarray = np.array([100.0, 60.0, 40.0]) # training window, 200 rowsprod: np.ndarray = np.array([56.0, 64.0, 80.0]) # this week, same bin edges
expected: np.ndarray = train / train.sum()actual: np.ndarray = prod / prod.sum()
psi: float = float(((actual - expected) * np.log(actual / expected)).sum())d_stat: float = float(np.abs(np.cumsum(actual) - np.cumsum(expected)).max())
print(round(psi, 2)) # -> 0.27print(round(d_stat, 2)) # -> 0.22Both printouts match the hand computation above: PSI 0.27 clears the 0.25 line, and the largest cumulative gap is 0.22 at the low bin.
Watch Out For
A pipeline change trips the alarm
Symptom: PSI spikes overnight across a dozen features right after a deploy, yet accuracy and the raw event counts look flat. Someone renamed a column, changed a unit, or swapped the normalization, so the engineered mix moved while the world stood still. Fix: version the reference with the pipeline, re-baseline the detector on the new pipeline's output, and keep the old reference around for audit. Never widen the threshold to quiet a deploy.
A tiny shift with a tiny p-value
Symptom: the KS p-value drops below 0.001 on 80,000 rows with and PSI at 0.01, and the pager fires over a move no customer would notice. Big samples make tiny gaps statistically certain without making them important. Fix: gate every alert on effect size first, PSI or , plus the feature's importance in the model, and treat the p-value as supporting evidence only. Batch the survivors into one daily summary.
The Quick Version
- Data drift detection compares production inputs against training inputs without needing fresh labels.
- Our three checkout bins move 100 60 40 to 56 64 80, scoring PSI 0.27, which clears the 0.25 real-shift line.
- The 100 60 40 to 56 64 80 checkout bins give KS , the largest gap between the two cumulative curves.
- PSI under 0.1 is quiet, 0.1 to 0.25 is worth a look, and over 0.25 starts the retrain conversation.
- Concept drift is the different failure where inputs look steady and the right answer moves, so it needs fresh labels while drift detection does not.