Feature Store

Training and serving often compute the same feature twice and drift apart. Define each feature once and let one store serve both paths.

One definition feeds both stores: Wednesday point-in-time count is 5 while a naive join reads 9, and 600 seconds offline against 0.02 seconds online.

Why Does This Exist?

Teams first write warehouse SQL for training, then rewrite it in serving code against a fast base. One side uses greater-or-equal, the other uses greater-than, and live counts drift by a few events. Accuracy falls with no failing test because each path looks right alone. That gap has a name: training-serving skew.

A second pain follows. Five teams each compute 30-day spend with slightly different windows, so the same user has five truths. This page covers one definition served twice. Row identity behind each build lives in data versioning, the scored runs that consume the rows live in experiment tracking, and the live path that reads the values lives in model serving.

Basics first: a feature is a named value per entity and time, like 30-day spend for user 1002. The offline store is history for training tables. The online store is latest values for millisecond reads. One definition compiles to both.

Think of It Like This

A central kitchen for food trucks

A taco chain runs five trucks. Each truck once mixed its own salsa with its own recipe card, so the same taco tasted different per corner. The chain opens one central kitchen that mixes salsa once and trucks it out daily. Every truck serves the same salsa because one card governs every batch.

That maps directly. Trucks are training and serving paths, salsa is the feature value, the recipe card is the single definition, and the daily delivery is the sync from offline history to the online base.

Where it stops: salsa tolerates small batch shifts, while models do not. A pinch more salt still sells, but a count of 9 where training saw 5 moves the score. The store must promise exact parity, not fresh-enough taste.

How It Actually Works

One definition compiles to two bases kept in sync by replay and streaming.

The two stores

The offline store keeps every past value for training joins, usually in columnar files. It answers bulk history slowly. The online store keeps only the latest value per key in a fast base and answers one row in milliseconds. Both read the same definition, so parity holds by construction rather than by matching two codebases.

Point-in-time correctness

Training must attach values as they stood at label time, not as they stand today. A naive join to current values leaks the future. A point-in-time join retrieves the value current at each event timestamp. Without it backtests look strong and live falls on day one.

Worked example: 5 against 9 with the same average

User 1002 triggers an event on Wednesday. In the prior 30 days they made 3 settled buys: 18.0, 32.0, and 40.0. The average is:

(18.0 + 32.0 + 40.0) / 3.0 = 90.0 / 3.0 = 30.00

Offline and online both serve 30.00 from the same definition. The offline build takes 600 seconds while the online read takes 0.02 seconds:

600 / 0.02 = 30000

Same number, 30000 times faster live. Counts tell the skew story. The Wednesday point-in-time count is 5 prior events. By the weekend the user made 4 more, so the current total reads 9. A naive training join to current values returns 9 and leaks:

9 - 5 = 4

Four future events in one training row. Repeat across thousands of rows and the model learns future knowledge that never exists live. That is the diagram above: 5 point-in-time against 9 naive, 30.00 on both paths, and 600 seconds against 0.02 seconds.

Code

The point is the parity check: prove both paths serve the same average from one tuple, with no second formula.

1purchases: tuple = (18.0, 32.0, 40.0)2
3offline_avg: float = (purchases[0] + purchases[1] + purchases[2]) / 3.04online_avg: float = (purchases[0] + purchases[1] + purchases[2]) / 3.05
6print(f"{offline_avg:.2f} {online_avg:.2f}")  # -> 30.00 30.007print(f"{9 - 5}")  # -> 48print(f"{600 / 0.02:.0f}")  # -> 30000

Both paths print 30.00 because both read the same tuple. The count gap prints 4 and the speed gap prints 30000. Change one buy to 50.0 and both averages move together, which a rewrite in a second language would break.

Watch Out For

Joining current values instead of point-in-time values

A training table joins to the latest snapshot without a time bound. Future buys leak into past rows, backtests glow, and live drops on launch.

Symptom: strong offline scores that vanish online, with event-time rows carrying weekend counts. Fix: build training rows with point-in-time joins at event time, and add a test that a Wednesday row returns 5 where the current snapshot reads 9.

Serving stale online values after a backfill

A definition fix rebuilds history but the online base keeps yesterday latest value. Offline improves while live stays flat.

Symptom: training metrics rise after a fix with no live move, and the online key timestamp predates the backfill. Fix: replay the backfill online, track row versions with data versioning, and watch key freshness where the model serves.

The Quick Version

Training-serving skew is the gap between warehouse logic for training and rewritten logic for live reads.
One definition compiles to an offline history base and an online latest-value base kept in sync.
Three buys of 18.0, 32.0, and 40.0 average to 30.00 on both paths from the same tuple.
A Wednesday event joins 5 prior events point-in-time, while a naive current join reads 9 and leaks 4 future events.
A 600 second offline build against a 0.02 second online read is a 30000 fold gap for the identical number.