Skip to content
AI360XpertBeta

ML System Design in Six Gated Phases

Interviews ask how to build a news feed ranker or a fraud filter. Six phases in fixed order turn that open question into a defensible answer where each decision gates the next.

Six gated phases run requirements, metric, data, model, serving and monitoring in order, and a spam filter at 6000 QPS with 100ms budget picks a small model plus a 0.20 PSI retraining rule
Six gated phases run requirements, metric, data, model, serving and monitoring in order, and a spam filter at 6000 QPS with 100ms budget picks a small model plus a 0.20 PSI retraining rule

Why Does This Exist?

A prompt says design a news feed ranker. The unprepared answer starts with a transformer and ends there. Then the interviewer asks about 100 millisecond latency, missing labels, and a feedback loop that promotes outrage, and the transformer answer has nowhere to go.

The miss is order, not knowledge. Latency kills half the models before training starts. Missing labels kill the rest. ML workflow names the lifecycle this design must live inside. This page covers the interview scaffold that forces requirements before metric, metric before data, data before model, model before serving, and everything before monitoring, so each choice is constrained by the one before it.

Think of It Like This

An architect hired for a school building

An architect starts with the brief: how many pupils, which site, which budget, which safety rules. Then comes structure, then materials, then the construction plan, then the maintenance contract. Nobody picks marble before knowing the site can carry it.

Brief maps to requirements, the success measure maps to metric, soil surveys map to data, material choice maps to model, construction maps to serving, and maintenance maps to monitoring. Each phase locks one decision the next phase must respect.

Where it stops: buildings stay put and maintenance means repairs. Model inputs shift weekly and predictions change user behavior, so monitoring here means scheduled retraining and rollback, not just fixing cracks.

How It Actually Works

Call the north star the one business number that defines success. Call the offline proxy the lab metric that predicts it. Call guardrails the numbers you promise not to harm. Each phase ends with a locked answer. An unlocked earlier phase reopens every phase after it.

The six gates in order

Requirements lock prediction, scale, latency, freshness and constraints. Metric locks north star, proxy and guardrails. Data locks labels, features available at prediction time, and batch versus streaming plumbing. Model locks a baseline first, then only the complexity the latency budget allows. Serving locks online, near real time or offline paths plus fallback and canary plan. Monitoring locks alert thresholds, drift checks and retraining triggers. The build behind phases three to five lives in ml pipeline architecture. Life after launch lives in model monitoring.

Worked example: a spam filter with numbers that bite

Input is mail body plus headers. Output is spam probability. Scale is 500 million mails a day. Seconds per day are 86,400, so average load is:

500,000,000/86,400≈5,787≈6,000 queries per second500{,}000{,}000 / 86{,}400 \approx 5{,}787 \approx 6{,}000 \text{ queries per second}

Latency must stay under 100 milliseconds so delivery never waits. Freshness needs weekly retraining because tactics shift daily. The costliest error is blocking legit mail, so the north star is false positive rate below 0.1 percent at 99 percent spam recall. Offline proxy is AUC on a held out labeled set. Guardrail is p99 inference under 80 milliseconds.

Labels come from mark as spam actions. Sender reputation must be computed as of delivery time, not today, or history leaks the future. Pipeline is streaming ingest with batch training on a 30 day window. Baseline is a rule filter, step two is logistic regression on text plus sender features, step three is boosted trees. Large transformers stop here because the 100 millisecond budget rules them out. Serving is online with dynamic batching, rule fallback when the endpoint is down, and canary 5 percent to 20 percent to 100 percent over 48 hours. Monitoring alerts when false positive rate tops 0.15 percent for 30 minutes and retrains when the sender domain PSI tops 0.20 or weekly, whichever comes first. That chain is the diagram: six boxes, 6,000 QPS under the first, 100 milliseconds under the fifth, 0.20 PSI under the sixth.

Why the baseline comes first

A simple floor tells you where the problem is hard. If rules plus regression already hit the false positive target, complexity buys risk without gain. Add weight only when the baseline fails in one measured way the next rung fixes.

Code

The point is the scale check: turn a daily volume into the QPS number that gates model and serving choices.

def qps_needed(mails_per_day: int, peak_factor: float = 1.5) -> str:    """Convert daily volume to average and peak queries per second."""    avg: float = mails_per_day / 86400    peak: float = avg * peak_factor    return f"avg {avg:.0f} peak {peak:.0f}"

print(qps_needed(500_000_000))  # -> avg 5787 peak 8681print(qps_needed(10_000_000))  # -> avg 116 peak 174

The printout matches the hand division: 500 million a day means about 5,787 average and 8,681 at 1.5 times peak, which is why the example rounds to 6,000 QPS when sizing batches and latency. A 10 million mail product needs only about 116 average, a different serving tier entirely.

Watch Out For

Choosing the model before the latency budget

A team proposes a 7 billion parameter model with 800 millisecond p99 for a filter that must answer in 100 milliseconds. Demos look smart and production queues overflow on day one.

Symptom: offline wins that cannot fit the serving budget, then a rushed rewrite under launch pressure. Fix: lock online versus batch and the millisecond budget in requirements, carry it into model choice, and stop the complexity ladder where ab testing for ML can still ship a canary safely.

No baseline, so no lesson when the big model wins

The team ships boosted trees directly, hits target, and nobody knows whether rules would have done. Six months later a simpler bug fix needs a model change and every edit risks the whole stack.

Symptom: upgrades with no ablation, where each release is all or nothing. Fix: ship the rule baseline first, record its false positive rate, then promote only on a measured gap. Experiment tracking keeps those runs comparable instead of remembered.

The Quick Version

  • Six phases run in order: requirements, metric, data, model, serving, monitoring, and each phase gates the next.
  • Our spam filter at 500 million mails a day means about 5,787 average QPS, rounded to 6,000 for sizing, with a 100 millisecond ceiling.
  • The north star is false positive rate below 0.1 percent at 99 percent recall, with p99 under 80 milliseconds as guardrail.
  • Baseline first, then complexity only as far as latency allows, with rule fallback and a 5 to 20 to 100 percent canary.
  • Monitoring closes the answer with named triggers, including retraining when PSI tops 0.20.