ML System Design Framework

Why Does This Exist?

ML system design interviews ask open-ended questions: "Design a news feed ranking system." "How would you build fraud detection at Stripe?" "Design a content moderation pipeline."

The trap is starting with the model. Candidates who jump straight to "I'd use a transformer" fail to notice that the problem might not need a model at all, or that the latency constraint makes transformer inference impossible, or that labels don't exist yet. The framework exists to prevent that mistake — every phase only makes sense after the previous one is resolved.

Think of It Like This

Phase 1 — Requirements

This is where most candidates lose marks because they assume too much. Spend five minutes here.

Functional: What exact prediction does the system make? What are the inputs and outputs? What does "correct" look like?

Scale: Queries per second, number of items in the catalogue, number of users. These determine infrastructure tier.

Latency: Is this real-time (online inference, P99 < 200ms) or batch (offline, hours acceptable)? This is often the most constraining requirement — it kills half the model architectures before you even start.

Freshness: How quickly do predictions need to reflect new events? A fraud model that can't adapt in minutes is useless; a movie recommender that updates daily is fine.

Constraints: Cold start handling, regulatory compliance, budget, team size.

Phase 2 — Metric

Pick the north star metric first. This is the business metric that defines success: CTR, conversion rate, false negative rate on fraud, task completion rate. Everything else is derived from it.

Then define the offline proxy you'll use during development (since you can't measure north star in a lab): NDCG@10 for ranking, AUC for classification, BLEU for translation. The proxy must correlate with the north star — if it doesn't, you'll optimise for the wrong thing.

Finally, define guardrail metrics: metrics you commit to not harming even if the north star improves. Latency, error rate, and fairness metrics are common guardrails.

Phase 3 — Data

Labels: Do they exist? How clean are they? What's the labelling strategy if they don't (implicit feedback, active learning, crowd-sourcing)?

Features: What signals are available at prediction time? Which of those are available at training time too? (Feature leakage is where most production failures originate — a feature that wasn't available at the moment of the historical decision.)

Pipeline: Is this streaming (Kafka, Flink) or batch (Spark, Airflow)? Does the pipeline need point-in-time correctness for training?

Phase 4 — Model

With requirements, metrics, and data resolved, model selection is constrained rather than arbitrary.

Start with a simple baseline. Logistic regression, a decision tree, or even a popularity-based heuristic. The baseline gives you a floor and teaches you where the problem is hard. Only add complexity when the baseline fails in a specific, measurable way.

Then go up the complexity ladder only as far as the latency and serving constraints allow.

Phase 5 — Serving

This is where requirements come back. The latency budget determines whether you can run a 7B parameter model online or whether you need to pre-compute and cache predictions offline.

Key decisions: online vs offline vs near-real-time inference, batching strategy, caching layer, fallback when the model is unavailable, and the AB testing plan for deploying the first version.

Phase 6 — Monitoring

Every system design answer should close with the monitoring plan. What metric triggers a retraining? How do you detect data drift? What's the rollback path if the model degrades?

A monitoring plan that references specific metrics (PSI > 0.20, online eval score < 3.8) is worth significantly more than "we'll add monitoring later."

Show Me the Code

The framework applied to a worked example — "Design a spam filter for email":

1Phase 1 — Requirements2  - Input: email body + headers. Output: spam / not spam probability.3  - Scale: 500M emails/day → 6,000/sec peak4  - Latency: < 100ms (must not delay email delivery)5  - Freshness: spammer tactics change daily → model must be retrained ≥ weekly6  - Constraint: false negative rate (spam reaches inbox) is more tolerable than7    false positive rate (legitimate email blocked)8
9Phase 2 — Metric10  - North star: false positive rate < 0.1% at 99% recall on spam11  - Offline proxy: AUC-ROC on held-out labelled set12  - Guardrail: p99 inference latency < 80ms13
14Phase 3 — Data15  - Labels: user "mark as spam" + "not spam" actions → implicit positive/negative16  - Leakage risk: sender reputation score must be computed as of delivery time, not today17  - Pipeline: streaming ingest, batch training on 30-day sliding window18
19Phase 4 — Model20  - Baseline: keyword/rule filter (instant, interpretable)21  - Step 2: logistic regression on tf-idf + sender features22  - Step 3: gradient boosted trees with embeddings for body text23  - Stop here: 100ms constraint rules out large transformers24
25Phase 5 — Serving26  - Online inference, model served via REST, dynamic batching for burst traffic27  - Fallback: rule-based filter if model endpoint is down28  - Canary: 5% → 20% → 100% over 48h, gated on false positive rate29
30Phase 6 — Monitoring31  - Alert: false positive rate > 0.15% sustained 30 min32  - Drift check: PSI on sender-domain distribution weekly33  - Retraining trigger: PSI > 0.20 OR weekly scheduled34  - Rollback: previous model version, < 60 second switch

Watch Out For

The Quick Version

• Six phases, in order: Requirements → Metric → Data → Model → Serving → Monitoring.
• Each phase gates the next: don't pick a model before you know the latency budget; don't design serving before you know what the model outputs.
• Always start with a simple baseline model. Justify complexity only when baseline fails in a specific measurable way.
• Close every system design with a monitoring plan that names specific metrics and retraining triggers.

What to Read Next

• ml-pipeline-architecture — The engineering backbone connecting the data, model, and serving phases.
• incident-response-for-ml — The postmortem and recovery procedures the monitoring phase feeds into.