Model Serving: Real-Time API vs Batch

Serving turns a trained model into something other systems can call: a real-time API that answers one request in milliseconds, or a batch job that scores millions of rows overnight and parks the answers in a table.

One request travels a real-time API in 20 ms for 50 requests per second, while a batch job scores 8 rows in 40 ms for 200 per second, trading wait for throughput.

Why Does This Exist?

A model in a notebook earns nothing. Value starts when other systems can ask it questions: the checkout page asking about fraud, the marketing tool asking who might churn. Serving is the engineering that makes those questions answerable safely and at scale.

Without it, teams fall back on fragile habits: a data scientist hand-running a script to email a CSV, or a web developer stuffing heavy model dependencies into the storefront app until it bloats and couples every deploy. Serving draws the boundary: the model lives behind an interface, and both sides can change independently. Once it serves, rollout strategy matters, which is where blue-green deployments and canary deployments come in, and live predictions need watching through model monitoring.

The one idea you need first

Latency is how long one answer takes. Throughput is how many answers finish per second. They trade against each other constantly: the fastest single answer rarely comes from the system with the highest total output. Every serving choice below is that trade with numbers on it.

Think of It Like This

A barista versus an overnight bakery

A barista makes your cup while you wait. One order in, one cup out, two minutes, and the whole setup exists to minimize your wait. An overnight bakery works while the city sleeps: thousands of loaves go into the ovens at midnight and land on store shelves by morning, and nobody stands at the counter waiting.

The mapping is direct. Your order is the request. The barista is the real-time API. The ovens running all night are the batch job. The store shelf is the predictions table the app reads without ever meeting the model.

Where it stops: the bakery cannot serve a customer in a hurry, and the barista cannot feed a wedding. Pick the one whose timing matches the need, because each fails absurdly at the other's job.

How It Actually Works

Serving splits into two patterns by one question: does anyone wait for this answer right now. A yes means real-time. A no means batch.

Real-time: one request, answered now

The model runs as a service behind a REST or gRPC endpoint. The client sends features, the server runs predict, and the answer returns in milliseconds. Because joins against a warehouse would blow the budget, features usually come from a fast lookup in feature stores and the preprocessing travels with the model artifact.

The fixture: one fraud check costs 20 ms end to end. Throughput is $1 / 0.020 = 50$ requests per second per worker. Need more and you add workers, which is simple and linear, but every millisecond of added feature fetching comes straight out of the user experience.

Batch: many rows, answered later

A scheduled job loads the model across workers, scans a warehouse table, scores every row, and writes the predictions back where apps can read them like any other data. In practice that job is declared as a step in pipeline orchestration, so a failed nightly run retries instead of restarting from scratch. The marketing tool never calls the model. It queries yesterday's scores.

The fixture: the same model scores 8 rows in 40 ms on one worker. Throughput is $8 / 0.040 = 200$ rows per second, four times the real-time rate, because batching amortizes loading and vectorizes the math. The price is wait: rows sit in a queue until the nightly run, then the run itself takes minutes to hours across the full table.

Worked example

Take a churn campaign over 200,000 users with the numbers above. Real-time at 50 per second needs $200{,}000 / 50 = 4{,}000$ seconds, over an hour of constant serving, to answer everyone one by one. Batch at 200 per second needs $200{,}000 / 200 = 1{,}000$ seconds, about 17 minutes, and the campaign tool reads a table the next morning. For a dashboard read daily, batch wins by a mile. For a checkout fraud check with a 100 ms budget, batch is disqualified no matter how cheap, because the answer arrives after the decision.

Code

The point of this snippet is the decision rule: turn two measured timings into throughput, then let the latency budget pick the pattern.

1def throughput(requests: int, seconds: float) -> float:2    """Answers per second for one worker."""3    return requests / seconds4
5
6def choose(latency_budget_ms: float) -> str:7    """Pick the pattern from the caller's patience."""8    if latency_budget_ms < 1000:9        return "real-time API"10    return "batch job"11
12
13single = throughput(1, 0.020)14batched = throughput(8, 0.040)15print(round(single, 1))  # -> 50.016print(round(batched, 1))  # -> 200.017print(choose(100))  # -> real-time API18print(choose(86_400_000))  # -> batch job

Fifty versus two hundred per worker is the trade, and the budget line makes the choice boring on purpose.

Watch Out For

Building a 50 ms API for a dashboard read each morning

Real-time serving buys availability, autoscaling, and latency monitoring, then charges for them forever. The symptom is a pager that wakes people for a lead-scoring API whose only reader opens a dashboard at 9 AM.

Fix it by defaulting to batch whenever the freshest acceptable answer is hours old. Promote to real-time only when a named decision truly waits on the answer, with its millisecond budget written down.

Training on one recipe, serving another

The notebook cleans text one way and the API cleans it slightly differently: a different tokenizer version, a missing imputation, a renamed column. The symptom is a model that validated beautifully and predicts garbage in production, with no error anywhere.

Fix it by shipping the preprocessing with the model as one versioned artifact and testing the served path on training rows before release. If training rows do not score identically through the API, the pipeline is broken, not the model.

The Quick Version

Serving exposes a trained model behind an interface so apps and models deploy independently.
Real-time answers one request now: 20 ms each, about 50 per second per worker, bought with serving infrastructure.
Batch scores millions later: 8 rows per 40 ms, about 200 per second per worker, bought with waiting.
The latency budget picks the pattern: a waiting decision means an API, a daily read means a table.
Ship preprocessing with the model artifact, or training-serving skew will rot predictions silently.