Concept Drift
The inputs look the same but the right answer changed, so a once accurate model starts missing. Concept drift is that broken link between features and labels.
Why Does This Exist?
Your fraud model earned its keep last year. Now the same features go in, yet the accounts it flags stopped matching the ones that actually default. Nothing broke in the pipeline. No column changed type. The world moved and left the model behind.
This page is about that specific break. When the inputs themselves shift, that is data drift, and data drift detection is the page that covers it. Here the inputs look identical and the link between inputs and labels quietly changes. You'll get the notation for that link, one loan example you can check by hand, and the monitoring habit that catches it live. The live part belongs to model monitoring, which is where this drift usually surfaces first.
Think of It Like This
The old fishing mark
You fish the same mark every spring: same GPS pin, same bait, same hour. For years it fills the boat. Then the current shifts and the fish hold somewhere else. Your sounder shows the same water and your log shows the same routine, but the catch collapses.
The GPS pin is your feature vector , the catch is the label , and the current is the relationship between them. Checking the sounder is watching : it tells you the water looks the same, which is true and useless. You don't need a new sounder. You need a new mark.
Where it stops: fish move on their own, and a new mark fixes the season. A drifted model keeps predicting with the old rule until you retrain it, and retraining on the wrong window can leave you worse off than the stale model.
How It Actually Works
Write for the features, for the label, for the chance of each label given the features, and for how the features themselves are distributed. Train time is and production is . Concept drift is the statement with roughly unchanged. Same inputs in, different right answers out, and an untouched model scoring worse.
The three shifts, side by side
- Covariate shift moves while stays put. Fresh inputs or reweighting usually fix it.
- Label shift moves while the boundary stays. Fix the threshold or the base rate.
- Concept drift moves itself. Only a new rule helps, which means fresh labels and retraining. The full taxonomy with the tests for each lives at data drift and distribution shift.
Worked example: a loan rule falls from 0.85 to 0.60
Take 20 applicants: 10 with low scores () and 10 with high scores (). The rule rejects low and accepts high, and means default.
At train time, 8 of the 10 low applicants default and 1 of the 10 high applicants defaults. So and . The rule gets 8 right in the low group and 9 right in the high group:
In production the mix is still 10 and 10, so never moved. But now only 3 of the 10 low applicants default, so while the high group stays at 0.1. Same rule, new world:
Accuracy fell 25 points with identical inputs and an untouched model. The diagram draws these exact counts: 8 of 10 and 9 of 10 on the left, 3 of 10 and 9 of 10 on the right.
Why fresh labels decide everything
You can't see move until some labels arrive. While you wait, watch the proxies: the mix of predictions shifts, confidence sags toward 0.5, and the flagged volume moves before confirmed accuracy does. Then settle it on a rolling window of recent labels and re-fit on a schedule, which is the model evaluation loop.
Two patterns show up again and again. Spam wording drifts a little every month until last year's rules miss, which is gradual drift. Fraud tactics or shopping habits flip within days of a breach or a lockdown, which is sudden drift. Either way the signature is the same: steady inputs, sagging accuracy, and a slice rate that moved.
Code
The point is the check: rebuild the two windows from counts and confirm the hand computation of 0.85 and 0.60.
import numpy as np
def accuracy(y_true: np.ndarray, y_pred: np.ndarray) -> float: return float((y_true == y_pred).mean())
y_train: np.ndarray = np.array([1] * 8 + [0] * 2 + [1] * 1 + [0] * 9)rule: np.ndarray = np.array([1] * 10 + [0] * 10)y_prod: np.ndarray = np.array([1] * 3 + [0] * 7 + [1] * 1 + [0] * 9)
print(f"{accuracy(y_train, rule):.2f}") # -> 0.85print(f"{accuracy(y_prod, rule):.2f}") # -> 0.60Both printouts match the hand computation above: 17 of 20 at train time, 12 of 20 after the shift.
Watch Out For
Treating it as a data problem
Symptom: the input dashboards glow amber, so you reweight, resample, or rebuild features, and accuracy stays flat. was never the problem. Fix: compare the label rate inside fixed feature slices, like low score applicants, across windows. If the slice rate moved while the slice size held, it is concept drift, so collect fresh labels and retrain instead of reshaping inputs.
Retraining on whatever arrived last
Symptom: drift trips the alarm and the next job retrains on the last few days, which may hold an attack burst, a holiday spike, or barely any labels. The new model ships worse than the stale one. Fix: mix recent data with a stable history window, hold out the newest week for the check in model evaluation, and promote the retrained model only when it beats the old one there.
The Quick Version
- Concept drift means changed: the same features now point at different labels, while the inputs can look identical.
- Our loan rule rejected low scores and accepted high ones, scoring 17 of 20 (0.85) at train time and 12 of 20 (0.60) after moved from 0.8 to 0.3.
- Data drift moves the inputs while concept drift moves the answers, so check the slice rates before touching the features.
- Without fresh labels, watch the proxies: the prediction mix and the flagged volume move before confirmed accuracy does.
- Retrain on a curated window of recent plus history data, and promote only the model that beats the old one on the newest labels.