Feature Stores

A feature store is a centralized repository that computes, stores, and serves ML features consistently between training and inference, eliminating the most common source of training-serving skew.

Why Does This Exist?

Every ML team eventually discovers that the feature pipeline used during training computes features differently from the one used during serving. The result: a model trained on one distribution served to another, with no alarm ringing. Feature stores solve this by making feature definitions the single source of truth, computed once and served consistently everywhere.

A secondary benefit: teams can share features. Instead of every team computing "user 30-day purchase total" independently with slightly different definitions, one feature in the store serves all teams.

Think of It Like This

A central ingredients depot for a restaurant chain

A restaurant chain doesn't let each branch prepare its own sauces from scratch with slightly different recipes. They have a central commissary kitchen that prepares standardized ingredients, then distributes them to every branch. Every burger tastes the same because the ingredients are the same. A feature store is the commissary kitchen for ML features: one place computes and stores them, and every model — during training and during serving — gets the same values.

How It Actually Works

The two stores

A feature store has two faces:

Offline store: A historical database of feature values, typically stored in a columnar format (Parquet, BigQuery, Redshift). Used during training to generate datasets by joining features with labels at specific points in time. Optimized for large batch reads, not low-latency access.

Online store: A low-latency key-value store (Redis, DynamoDB, Bigtable) that materializes the latest feature values for entities. At inference time, a model looks up features for a user or product in milliseconds. Optimized for point lookups, not historical joins.

Point-in-time correctness

The most subtle feature store concept. When training a model to predict whether a user will churn, you need to know what features looked like at the time of the label, not today. A naive join would give you today's features for a label from six months ago — leaking the future into training. Point-in-time joins (also called "as-of" joins) retrieve the feature value that was current at each label's timestamp.

1Label: User X churned on 2024-03-152Feature join: retrieve user X's features as of 2024-03-14 (one day before label)3Result: no future leakage

Code

1# Defining and materializing features with Feast2from feast import Entity, Feature, FeatureView, FileSource, ValueType3from datetime import timedelta4
5# Define the entity6user = Entity(name="user_id", value_type=ValueType.INT64)7
8# Define the data source9user_stats_source = FileSource(10    path="data/user_stats.parquet",11    event_timestamp_column="event_timestamp",12)13
14# Define the feature view15user_stats_view = FeatureView(16    name="user_stats",17    entities=["user_id"],18    ttl=timedelta(days=90),19    features=[20        Feature(name="purchase_count_30d", dtype=ValueType.INT64),21        Feature(name="total_spend_30d", dtype=ValueType.FLOAT),22        Feature(name="days_since_last_order", dtype=ValueType.INT64),23    ],24    source=user_stats_source,25)26
27# At training time: get historical features (point-in-time correct)28from feast import FeatureStore29store = FeatureStore(repo_path=".")30
31training_df = store.get_historical_features(32    entity_df=labels_df,  # has user_id and event_timestamp columns33    features=["user_stats:purchase_count_30d", "user_stats:total_spend_30d"],34).to_df()35
36# At serving time: get online features (millisecond latency)37online_features = store.get_online_features(38    features=["user_stats:purchase_count_30d", "user_stats:total_spend_30d"],39    entity_rows=[{"user_id": 1001}, {"user_id": 1002}],40).to_dict()

Watch Out For

Feature freshness vs. cost trade-off

Online stores need to be kept fresh — materialization jobs run on a schedule (hourly, daily, real-time streaming). Fresher features cost more (compute, storage, streaming infrastructure). Model performance often degrades gracefully with stale features; profile before over-engineering freshness.

Point-in-time correctness violations

If your training pipeline joins features on user_id without a timestamp constraint, you're silently leaking future information into training. The model will overfit to information it wouldn't have at inference time. Always use point-in-time joins when training on historical data with temporal labels.

The Quick Version

A feature store is a centralized system for computing, storing, and serving ML features consistently at training and serving time.
The offline store holds historical feature values for training; the online store serves low-latency features at inference.
Point-in-time joins prevent label leakage by retrieving features as they existed at the label timestamp.
Feature stores also enable feature reuse across teams, reducing duplicated computation and inconsistent definitions.