Core ML

Batch vs Real-Time Inference

Do you need the model to give you an answer right this second, or can you just let it crunch all the data overnight while you sleep? The choice fundamentally changes how you build your infrastructure.

Real-Time inference optimizes for Latency, predicting on single inputs instantly via REST APIs. Batch inference optimizes for Throughput, loading thousands of rows at once overnight and saving the results to a database.

Why Does This Exist?

When deploying a machine learning model, the hardest architectural decision you have to make is how users will consume the predictions. This boils down to a fundamental tradeoff between Latency (how fast you get a single answer) and Throughput (how many total answers you can get per hour).

If you are building a self-driving car, you need the model to detect pedestrians in 20 milliseconds. This requires Real-Time Inference.
If you are building a system that predicts which of your 10 million customers will churn this month, you don't need the answer in milliseconds. You can run the model overnight on all 10 million customers at once. This is Batch Inference.

Think of It Like This

Imagine a ferry crossing a river.

Real-Time Inference is like a high-speed speedboat taxi. It only holds one person, but it takes them across the river instantly, exactly when they ask. It optimizes for low latency, but is terrible at moving 10,000 people.

Batch Inference is like a massive cargo ship. It waits at the dock until 10,000 people have boarded, and then slowly crosses the river. The person who boarded first had to wait hours for the ship to leave, but the ship successfully transported 10,000 people in a single trip. It optimizes for massive throughput.

1. Real-Time (Online) Inference

In real-time inference, the model generates predictions synchronously in response to an immediate user action.

How it works: The model is usually wrapped in a REST API or a gRPC service. The user clicks a button, their phone sends a request to the server, the model runs, and the server sends the result back immediately.
The Optimization Goal: Latency. You want the time-to-first-token or total response time to be as low as physically possible.
The Catch: Because you are processing requests one at a time as they arrive, GPU utilization is terrible. GPUs are designed to process massive grids of data in parallel, not single rows. Real-time inference is mathematically the most expensive way to run a model per-prediction.

2. Batch (Offline) Inference

In batch inference, you gather massive amounts of data and process it all at once, usually on a recurring schedule (e.g., every night at 2:00 AM).

How it works: You don't use a REST API. Instead, an orchestrator (like Apache Airflow) triggers a Python script. The script reads 10 million rows from a Data Warehouse (like Snowflake), feeds them into the model in massive chunks (e.g., 4,096 rows at a time), and writes the 10 million predictions back into the database.
The Optimization Goal: Throughput. You don't care if the first row takes 10 minutes to process, as long as the total 10 million rows finish before the morning.
The Catch: The predictions are always slightly stale. If a customer changes their behavior at 9:00 AM, the database won't reflect their new churn prediction until the batch runs again the next night.

The Hybrid: Streaming Inference

There is a middle ground. What if you need predictions fast, but you also need high throughput for thousands of events per second (like credit card fraud detection)?

You use Streaming Inference. Instead of a REST API, you connect your model to a message broker like Apache Kafka. As credit card swipes happen globally, they are pushed into a Kafka stream. The ML model constantly pulls small batches (e.g., 50 swipes) from the stream every few milliseconds, predicts fraud, and pushes the results to another stream. It offers near real-time latency with much better throughput than a REST API.

Watch Out For

Defaulting to Real-Time. Engineers love building REST APIs, so they default to real-time inference for every project. This is almost always a financial mistake. If the product feature does not physically require sub-second latency (e.g., generating a weekly personalized newsletter), you should always use Batch Inference. Batch inference is radically cheaper, easier to monitor, completely immune to traffic spikes, and never causes the user to stare at a loading spinner.

The Quick Version

Real-Time Inference uses REST APIs to serve predictions instantly. It optimizes for Latency but is expensive because it under-utilizes the GPU.
Batch Inference runs offline (usually overnight) on massive datasets. It optimizes for Throughput and is the cheapest, most efficient way to run a model.
Streaming Inference uses message queues (like Kafka) to process continuous streams of data, offering a balance of low latency and high throughput.
Never build a real-time system if a batch system will solve the business problem.