Skip to content
AI360XpertBeta

Concept Drift

The inputs look the same but the right answer changed, so a once accurate model starts missing. Concept drift is that broken link between features and labels.

A fixed loan rule scores 0.85 at train time and 0.60 after P(Y|X) shifts while P(X) stays put
A fixed loan rule scores 0.85 at train time and 0.60 after P(Y|X) shifts while P(X) stays put

Why Does This Exist?

Your fraud model earned its keep last year. Now the same features go in, yet the accounts it flags stopped matching the ones that actually default. Nothing broke in the pipeline. No column changed type. The world moved and left the model behind.

This page is about that specific break. When the inputs themselves shift, that is data drift, and data drift detection is the page that covers it. Here the inputs look identical and the link between inputs and labels quietly changes. You'll get the notation for that link, one loan example you can check by hand, and the monitoring habit that catches it live. The live part belongs to model monitoring, which is where this drift usually surfaces first.

Think of It Like This

The old fishing mark

You fish the same mark every spring: same GPS pin, same bait, same hour. For years it fills the boat. Then the current shifts and the fish hold somewhere else. Your sounder shows the same water and your log shows the same routine, but the catch collapses.

The GPS pin is your feature vector XX, the catch is the label YY, and the current is the relationship P(Y∣X)P(Y|X) between them. Checking the sounder is watching P(X)P(X): it tells you the water looks the same, which is true and useless. You don't need a new sounder. You need a new mark.

Where it stops: fish move on their own, and a new mark fixes the season. A drifted model keeps predicting with the old rule until you retrain it, and retraining on the wrong window can leave you worse off than the stale model.

How It Actually Works

Write XX for the features, YY for the label, P(Y∣X)P(Y|X) for the chance of each label given the features, and P(X)P(X) for how the features themselves are distributed. Train time is t=0t = 0 and production is t=1t = 1. Concept drift is the statement P0(Y∣X)≠P1(Y∣X)P_0(Y|X) \ne P_1(Y|X) with P(X)P(X) roughly unchanged. Same inputs in, different right answers out, and an untouched model scoring worse.

The three shifts, side by side

  • Covariate shift moves P(X)P(X) while P(Y∣X)P(Y|X) stays put. Fresh inputs or reweighting usually fix it.
  • Label shift moves P(Y)P(Y) while the boundary stays. Fix the threshold or the base rate.
  • Concept drift moves P(Y∣X)P(Y|X) itself. Only a new rule helps, which means fresh labels and retraining. The full taxonomy with the tests for each lives at data drift and distribution shift.

Worked example: a loan rule falls from 0.85 to 0.60

Take 20 applicants: 10 with low scores (X=lowX = \text{low}) and 10 with high scores (X=highX = \text{high}). The rule ff rejects low and accepts high, and Y=1Y = 1 means default.

At train time, 8 of the 10 low applicants default and 1 of the 10 high applicants defaults. So P0(Y=1∣X=low)=8/10=0.8P_0(Y = 1|X = \text{low}) = 8/10 = 0.8 and P0(Y=1∣X=high)=1/10=0.1P_0(Y = 1|X = \text{high}) = 1/10 = 0.1. The rule gets 8 right in the low group and 9 right in the high group:

accuracy0=8+920=1720=0.85\text{accuracy}_0 = \frac{8 + 9}{20} = \frac{17}{20} = 0.85

In production the mix is still 10 and 10, so P(X)P(X) never moved. But now only 3 of the 10 low applicants default, so P1(Y=1∣X=low)=3/10=0.3P_1(Y = 1|X = \text{low}) = 3/10 = 0.3 while the high group stays at 0.1. Same rule, new world:

accuracy1=3+920=1220=0.60\text{accuracy}_1 = \frac{3 + 9}{20} = \frac{12}{20} = 0.60

Accuracy fell 25 points with identical inputs and an untouched model. The diagram draws these exact counts: 8 of 10 and 9 of 10 on the left, 3 of 10 and 9 of 10 on the right.

Why fresh labels decide everything

You can't see P(Y∣X)P(Y|X) move until some labels arrive. While you wait, watch the proxies: the mix of predictions shifts, confidence sags toward 0.5, and the flagged volume moves before confirmed accuracy does. Then settle it on a rolling window of recent labels and re-fit on a schedule, which is the model evaluation loop.

Two patterns show up again and again. Spam wording drifts a little every month until last year's rules miss, which is gradual drift. Fraud tactics or shopping habits flip within days of a breach or a lockdown, which is sudden drift. Either way the signature is the same: steady inputs, sagging accuracy, and a slice rate that moved.

Code

The point is the check: rebuild the two windows from counts and confirm the hand computation of 0.85 and 0.60.

import numpy as np

def accuracy(y_true: np.ndarray, y_pred: np.ndarray) -> float:    return float((y_true == y_pred).mean())

y_train: np.ndarray = np.array([1] * 8 + [0] * 2 + [1] * 1 + [0] * 9)rule: np.ndarray = np.array([1] * 10 + [0] * 10)y_prod: np.ndarray = np.array([1] * 3 + [0] * 7 + [1] * 1 + [0] * 9)
print(f"{accuracy(y_train, rule):.2f}")  # -> 0.85print(f"{accuracy(y_prod, rule):.2f}")  # -> 0.60

Both printouts match the hand computation above: 17 of 20 at train time, 12 of 20 after the shift.

Watch Out For

Treating it as a data problem

Symptom: the input dashboards glow amber, so you reweight, resample, or rebuild features, and accuracy stays flat. P(X)P(X) was never the problem. Fix: compare the label rate inside fixed feature slices, like low score applicants, across windows. If the slice rate moved while the slice size held, it is concept drift, so collect fresh labels and retrain instead of reshaping inputs.

Retraining on whatever arrived last

Symptom: drift trips the alarm and the next job retrains on the last few days, which may hold an attack burst, a holiday spike, or barely any labels. The new model ships worse than the stale one. Fix: mix recent data with a stable history window, hold out the newest week for the check in model evaluation, and promote the retrained model only when it beats the old one there.

The Quick Version

  • Concept drift means P(Y∣X)P(Y|X) changed: the same features now point at different labels, while the inputs can look identical.
  • Our loan rule rejected low scores and accepted high ones, scoring 17 of 20 (0.85) at train time and 12 of 20 (0.60) after P(default∣low)P(\text{default}|\text{low}) moved from 0.8 to 0.3.
  • Data drift moves the inputs while concept drift moves the answers, so check the slice rates before touching the features.
  • Without fresh labels, watch the proxies: the prediction mix and the flagged volume move before confirmed accuracy does.
  • Retrain on a curated window of recent plus history data, and promote only the model that beats the old one on the newest labels.