Online / Offline Skew

The model is fine and the data is fine. The same input still produces two different feature vectors, because two separate code paths compute it.

One raw order travels the batch SQL path and the serving API path and comes out as two different feature vectors, so only a side-by-side diff on the same entities exposes the gap.

Why Does This Exist?

Your model validated at 0.88 and serves at 0.79. The weights are untouched, the input distribution has not moved, nothing is null, and no alert has fired. You retrain, and the new model validates at 0.88 again, then serves at 0.79 again.

Here is the usual cause. Training features were computed in SQL against the warehouse. Serving features were rewritten in the API language weeks later, by someone else, from the same ticket. One path treats a missing rating as zero and the other substitutes the global average. One closes a 30-day window exclusively and the other inclusively. Each path is self-consistent, each passes its own tests, and the same customer gets two different vectors depending on who asks.

This is the most common silent failure in applied ML, and it stays silent because nothing errors: no null, no shape mismatch, no exception. The model never sees a bug. From its side it is simply being asked about a slightly different customer than the one it trained on.

This page covers the disagreement between two code paths computing the same features. Its neighbours cover the lookalikes: data leakage is training seeing what it should never have seen, and data drift is the world itself moving after deployment. Skew is neither. The data is fine and the world stood still; the two implementations disagree.

The one idea you need first: a feature vector is the ordered list of numbers the model actually reads for one entity, like [amount, prior orders, average rating]. Everything the model knows about a customer passes through that list, so two lists for one customer means two different customers as far as the model is concerned. How those lists get built at scale is what training data pipelines describe.

Think of It Like This

Two kitchens cooking from the same recipe card

A chef develops a dish and writes the card: 200 grams flour, 4 grams salt, 30 minutes in the oven.

The second kitchen scales it for forty covers. Salt is measured there in teaspoons, so 40 grams becomes about seven teaspoons, and seven teaspoons is actually 42 grams. Their oven runs 15 degrees hot. Their 30 minutes start when the tray goes in, not when the oven reaches temperature.

Every one of those choices is defensible. Nobody made a mistake anyone would write up. And the plate that leaves the second kitchen is not the dish the chef signed off, so guests review a recipe that was never tested.

Where it stops: kitchens share taste and judgement, while code paths share nothing unless you force them to. Two cooks converge by talking; two implementations diverge in silence, which is why the fix here is structural (one implementation, two callers) rather than a better recipe card.

How It Actually Works

Write the two paths as functions of the same raw event $e$ . Training computes $x_{train} = f_{train}(e)$ and serving computes $x_{serve} = f_{serve}(e)$ . Skew is the statement that $x_{train} \neq x_{serve}$ for some real $e$ , where $x$ is the feature vector, $f_{train}$ is the batch implementation, and $f_{serve}$ is the serving implementation. Detection is therefore one operation: evaluate both functions on the same entities and count disagreements per column. Prevention is one structure: a single $f$ with two callers, which is the actual reason feature stores exist.

The four ways the paths diverge

Two implementations. The feature is written once for training and again for serving, in a different language, against a different store. Rounding, null defaults, timezone handling, units, string normalisation, and tie-breaking all get to differ quietly. This is the most frequent cause by far.

Different freshness. Training reads a table hours after the fact, after late events have landed. Serving reads it in the moment, before they have. Same query, same code, different answer, and the training-time value is one that could never exist at prediction time.

Different windows. A nightly batch closes "last 30 days" at midnight while serving closes it at now. For a customer who ordered this morning those differ, and the gap is largest for exactly the active customers you care about most. Streaming systems formalise this with watermarks.

Recomputed transform state. A scaler or encoder refit at serving time instead of loaded from the trained artefact. Now one request scores differently depending on which requests share its batch, which is not reproducible even in principle. Feature scaling covers the fitting side of this trap.

Skew is not leakage

Worth separating cleanly, because the symptom is identical and the fix is not. Leakage is training seeing something it should not have seen. Skew is serving computing something differently. Both give you a validation score production never reproduces. Ask which side is wrong: if the training feature could not have been computed at prediction time, that is leakage. If it could have been, but serving computes it another way, that is skew.

Worked example

One order. Three features: amount in pounds, orders in the prior 30 days, and the customer average rating, where this customer has never rated anything.

The batch path closes the window exclusively and defaults a missing rating to 0.0, giving $[42.0, 3, 0.0]$ . The serving path closes it inclusively, picking up an order placed exactly on the boundary, and defaults a missing rating to the global mean of 3.5, giving $[42.0, 4, 3.5]$ . Two of three columns differ.

Push both through one logistic model with weights $w = [0.01, 0.3, -0.2]$ . The batch logit is $0.01 \times 42 + 0.3 \times 3 - 0.2 \times 0 = 1.32$ , and $\sigma(1.32) = 0.789$ . The serving logit is $0.42 + 1.2 - 0.7 = 0.92$ , and $\sigma(0.92) = 0.715$ . Seven points of probability from two defaults and one boundary rule, each reasonable in isolation, and neither path contains a bug you could find by reading it alone.

Code

1import numpy as np2
3w: np.ndarray = np.array([0.01, 0.3, -0.2])  # amount, prior orders, avg rating4
5def score(features: np.ndarray) -> float:6    """One model, identical weights. Only the input differs."""7    return float(1 / (1 + np.exp(-(w @ features))))8
9batch: np.ndarray = np.array([42.0, 3.0, 0.0])    # exclusive window, missing -> 0.010serving: np.ndarray = np.array([42.0, 4.0, 3.5])  # inclusive window, missing -> 3.511
12print((batch != serving).tolist())              # -> [False, True, True]13print(round(score(batch), 3))                   # -> 0.78914print(round(score(serving), 3))                 # -> 0.71515print(round(score(batch) - score(serving), 3))  # -> 0.074

The first printed line is the whole detection strategy in miniature: run both paths on the same entity and compare. Everything below it is what the gap costs when you do not.

Watch Out For

A serving port reviewed by eye instead of diffed on data

The ticket says port orders_prior_30d to the serving path. Someone reads the SQL, writes the equivalent, tests it on a handful of customers, and the numbers look right.

Unit tests on both sides then pass, and they would pass even with the bug present, because each suite asserts its own path is self-consistent, which is exactly what skew preserves. Reading the two implementations side by side fails too, since the difference is usually a boundary condition or a default neither file states outright.

There is one check that works: run both paths on the same production entities, at volume, and compare outputs. A handful of customers will not do it, because the divergence is often conditional: only rows with a null, only orders on a window edge, only events near a daylight-saving change. Make it a CI test over a sample of real entities and treat any nonzero diff as a failure, not as drift.

Calling fit anywhere in the serving path

StandardScaler().fit_transform(batch) inside the request handler looks like the line that ran in training. It is a different operation.

Now the scaling depends on the batch. The same customer scores one way beside ordinary customers and another way beside one large account, because the batch mean moved. The prediction is no longer a function of its input, so replaying the request alone gives a third answer and the bug is not reproducible on demand.

The categorical version is quieter: an encoder rebuilt from the current batch assigns fresh integers, so column 4 means one category in training and another at serving. Every fitted parameter is part of the model artefact. Serialise the fitted transformer with the weights, load it, and call transform only.

The Quick Version

The model and the data are both fine. Two code paths compute the same feature differently.
There is no error, no null, and no alert. The only metric that catches it is a cross-path diff most teams never compute.
Four causes in rough frequency order: duplicated implementations, different freshness, different window edges, recomputed transform state.
One implementation with two callers is the fix, and it is the real reason feature stores exist.
Leakage is training seeing what it should not. Skew is serving computing differently. Same symptom, different repair.
Log the serving vector, replay the entity through the training pipeline, and alert on the per-feature diff rate rather than on accuracy.
Two defaults and one boundary rule moved a prediction from 0.789 to 0.715 on a single order.
If fit appears anywhere in your serving path, that is the bug.