Skip to content
AI360XpertBeta

Data Drift Detection

Today's inputs no longer look like training data, yet the model keeps answering with full confidence. Data drift detection compares the two windows bin by bin and raises the alarm before accuracy slides.

Three checkout bins move from 100 60 40 at training to 56 64 80 in production, scoring PSI 0.27 and KS D 0.22 over the 0.25 retrain line
Three checkout bins move from 100 60 40 at training to 56 64 80 in production, scoring PSI 0.27 and KS D 0.22 over the 0.25 retrain line

Why Does This Exist?

A model keeps answering after the world moves. It doesn't crash or complain. It just applies last year's pattern to this year's crowd, and the damage shows up weeks before the labels that would prove it.

That silence is the whole problem. Fresh labels rarely arrive in time, but fresh inputs arrive every hour. So you compare today's feature mix against the training mix, one feature at a time, and you'll set the alarm before accuracy slides.

This page covers that comparison for input drift: the PSI sum, the KS check, and one checkout example you can verify by hand. When the right answer itself moves while the inputs look steady, that is concept drift. The wider split of all three shifts lives at data drift and distribution shift, and the loop that acts on the alarm is model monitoring.

One line of grounding before the math. Call training time t=0t = 0 with distribution P0P_0, and production t=1t = 1 with P1P_1. Data drift is the claim that P1(X)P_1(X) no longer matches P0(X)P_0(X), where XX means the feature values you feed the model.

Think of It Like This

The coin jar on the counter

You keep a reference jar of 200 coins by the till: 100 small, 60 medium, 40 large. Every evening you scoop that day's 200 coins and sort them into the same three slots. One week the scoop reads 56, 64, 80. Same 200 coins, visibly different mix, and you know the till drifted before the books say so.

The jar is the training window, the evening scoop is the production window, and the three slots are the fixed bins. The PSI score is just the weighted surprise across those slots, with the slot that doubled shouting the loudest.

Where it stops: coins sit still while you count them, and you can see every coin. Model inputs are invisible at scale, which is why the detector is your only eyes. And coins never change what they are worth, so this jar says nothing about the day prices themselves move.

How It Actually Works

Fix the bins from training data and never move them. Write eie_i for the expected share of bin ii at training, meaning the fraction of training rows that fell there, and aia_i for the actual share of the same bin in production. Both lists sum to 1. The population stability index adds up each bin's surprise:

PSI=∑i(ai−ei) ln⁡ ⁣aiei\text{PSI} = \sum_i (a_i - e_i) \, \ln\!\frac{a_i}{e_i}

Here ln⁡\ln is the natural log. A bin that keeps its share contributes zero. A bin that doubles contributes a lot, because the log of the ratio grows while the gap in front of it grows too.

The usual reading is under 0.1 quiet, 0.1 to 0.25 worth a look, and over 0.25 a real shift that starts the retrain conversation. Always compare against training data, not last week, or slow drift becomes the new normal without ever tripping the wire.

Worked example: three bins score PSI 0.27

One checkout feature, amount per order, cut into three fixed bins: low, mid, high. The training window holds 200 orders: 100 low, 60 mid, 40 high. So e=[0.50,0.30,0.20]e = [0.50, 0.30, 0.20]. This week's 200 orders read 56 low, 64 mid, 80 high. So a=[0.28,0.32,0.40]a = [0.28, 0.32, 0.40].

Bin by bin, with the log to four places:

  • Low: (0.28−0.50)×ln⁡(0.28/0.50)=(−0.22)×(−0.5798)=0.1276(0.28 - 0.50) \times \ln(0.28/0.50) = (-0.22) \times (-0.5798) = 0.1276
  • Mid: (0.32−0.30)×ln⁡(0.32/0.30)=(0.02)×(0.0645)=0.0013(0.32 - 0.30) \times \ln(0.32/0.30) = (0.02) \times (0.0645) = 0.0013
  • High: (0.40−0.20)×ln⁡(0.40/0.20)=(0.20)×(0.6931)=0.1386(0.40 - 0.20) \times \ln(0.40/0.20) = (0.20) \times (0.6931) = 0.1386

Add them:

PSI=0.1276+0.0013+0.1386=0.2675≈0.27\text{PSI} = 0.1276 + 0.0013 + 0.1386 = 0.2675 \approx 0.27

That clears the 0.25 line, so it is a real shift, not noise. Notice the shape of it. The high bin doubled from 0.20 to 0.40 and supplied 0.1386, over half the total. The low bin nearly halved and supplied most of the rest. The mid bin barely moved and contributes almost nothing. PSI is dominated by proportional moves in a single bin, which is usually what you want to catch.

The KS check on the same bins

The Kolmogorov-Smirnov check asks a simpler question: what is the largest gap DD between the two cumulative curves, where each curve just adds the shares up to that bin. Write F0F_0 for the training cumulative and F1F_1 for production:

  • Training cumulative: [0.50,0.80,1.00][0.50, 0.80, 1.00]
  • Production cumulative: [0.28,0.60,1.00][0.28, 0.60, 1.00]
  • Gaps: [0.22,0.20,0.00][0.22, 0.20, 0.00], so D=0.22D = 0.22

DD runs from 0 (identical curves) to 1 (no overlap). Here 0.22 on 200 rows per side is a wide gap, and it points at the same low bin as the PSI sum. The two tests agree, which is the outcome you trust. When they disagree, believe the effect size over any p-value, because a p-value shrinks with sample size and will eventually flag a move too small to matter. If this shift holds, the fix is domain adaptation or a retrain, not a wider threshold.

Code

The point is the check: rebuild the two windows from counts and confirm the hand computation of 0.27 and 0.22.

import numpy as np
train: np.ndarray = np.array([100.0, 60.0, 40.0])  # training window, 200 rowsprod: np.ndarray = np.array([56.0, 64.0, 80.0])  # this week, same bin edges
expected: np.ndarray = train / train.sum()actual: np.ndarray = prod / prod.sum()
psi: float = float(((actual - expected) * np.log(actual / expected)).sum())d_stat: float = float(np.abs(np.cumsum(actual) - np.cumsum(expected)).max())
print(round(psi, 2))  # -> 0.27print(round(d_stat, 2))  # -> 0.22

Both printouts match the hand computation above: PSI 0.27 clears the 0.25 line, and the largest cumulative gap is 0.22 at the low bin.

Watch Out For

A pipeline change trips the alarm

Symptom: PSI spikes overnight across a dozen features right after a deploy, yet accuracy and the raw event counts look flat. Someone renamed a column, changed a unit, or swapped the normalization, so the engineered mix moved while the world stood still. Fix: version the reference with the pipeline, re-baseline the detector on the new pipeline's output, and keep the old reference around for audit. Never widen the threshold to quiet a deploy.

A tiny shift with a tiny p-value

Symptom: the KS p-value drops below 0.001 on 80,000 rows with D=0.02D = 0.02 and PSI at 0.01, and the pager fires over a move no customer would notice. Big samples make tiny gaps statistically certain without making them important. Fix: gate every alert on effect size first, PSI or DD, plus the feature's importance in the model, and treat the p-value as supporting evidence only. Batch the survivors into one daily summary.

The Quick Version

  • Data drift detection compares production inputs against training inputs without needing fresh labels.
  • Our three checkout bins move 100 60 40 to 56 64 80, scoring PSI 0.27, which clears the 0.25 real-shift line.
  • The 100 60 40 to 56 64 80 checkout bins give KS D=0.22D = 0.22, the largest gap between the two cumulative curves.
  • PSI under 0.1 is quiet, 0.1 to 0.25 is worth a look, and over 0.25 starts the retrain conversation.
  • Concept drift is the different failure where inputs look steady and the right answer moves, so it needs fresh labels while drift detection does not.