Cost Optimization in ML

Serving pays for GPUs even when traffic sleeps. Match replicas to demand so p95 stays flat while idle hours drop toward zero.

Three replicas by day and one by night hold p95 at 180 ms while GPU hours fall from 96 to 40, saving 56.

Why Does This Exist?

Inference bills arrive every month while training bills arrive once. A team launches 4 GPU instances for peak traffic, then watches them sit near 12 percent use overnight. Nothing is wrong with the model. The platform pays for capacity nobody uses.

This is a platform problem, not a modeling problem. The fix lives in instance choice, replica count, scaling policy, and where interruptible capacity is safe. This page covers that platform layer. The serving patterns live in model serving, the packaging that makes replicas identical lives in containerization for ML, and the use and latency signals that drive scaling live in model monitoring. A shared feature store can also cut repeated compute by serving one definition to every model.

Basics first: throughput is requests per second. Latency is seconds per request, usually p95, meaning 95 percent of requests finish faster than that number. Cost work keeps p95 flat while lowering GPU hours behind it. If p95 rises, the change was a slowdown, not a saving.

Think of It Like This

A shuttle bus line, not a parked fleet

A hotel runs shuttles to the airport. At noon it needs 3 buses. At midnight it needs 1. Owning 4 buses and running all 4 all day guarantees nobody waits, and it burns fuel all night moving empty seats.

The fix is not a smaller bus. It is a timetable: run 3 at noon, 1 at midnight, park the rest, and rent one cheap spare for the predictable tour trip that can wait if the rental is recalled.

That maps directly. Buses are GPU replicas, the timetable is the autoscaling policy, parking is scale to zero for batch work, and the cheap rental is spot capacity for jobs that checkpoint and resume.

Where it stops: a late bus annoys a traveler, while a missing replica breaches a latency objective and drops paid traffic. Buses can run late. Serving cannot miss p95 without cost.

How It Actually Works

Four levers, pulled in order. Each one lowers GPU hours without moving p95.

Right-sizing the instance

Pick the smallest instance that still meets p95. Profile memory and per-request time, then choose. A 7-billion-parameter model in 16-bit needs about 14 GB for weights alone, so it cannot fit a 16 GB card once activations land. A distilled 500-million-parameter model fits a small GPU at one quarter the hourly rate, and serving it there at 60 percent use beats serving it on a large card at 15 percent use.

Autoscaling with demand

A target-tracking policy watches queue length or p95 and adds replicas before latency slips. Scale to zero matters most for batch and async endpoints: zero replicas when idle means nights and weekends cost nothing. Keep one replica of headroom so a single slow replica does not breach the objective.

Mixing spot and committed capacity

On-demand covers live serving. Spot covers retraining, batch scoring, and sweeps that resume from a checkpoint. Committed use covers the steady baseline that runs all month. The mix is reviewed monthly because traffic shape drifts.

Shrinking the model to drop a tier

Quantization and distillation cut memory and per-request time, which moves work from a large GPU to a small one or from GPU to CPU. The accuracy trade lives in serving. The savings land here.

Worked example: 3 by day, 1 by night

One replica sustains 20 requests per second with p95 at 180 ms against a 200 ms objective. Day peaks at 50 requests per second. Night falls to 12 requests per second.

Day replicas needed:

\lceil 50 / 20 \rceil = 3

Night replicas needed:

\lceil 12 / 20 \rceil = 1

Three replicas carry 60 requests per second for 50 of demand. One carries 20 for 12 of demand. Both hold p95 at 180 ms.

Flat fleet over 24 hours:

4 \times 24 = 96

Scaled fleet with 3 replicas for 8 peak hours and 1 for 16 off-peak hours:

3 \times 8 + 1 \times 16 = 24 + 16 = 40

Savings:

96 - 40 = 56

Fifty-six GPU hours per day vanish while p95 stays at 180 ms. That is the diagram above: the same latency bar, two replica counts, and a cost block cut from 96 to 40.

Code

The point is the policy check: given traffic and per-replica capacity, compute replicas and daily GPU hours for flat versus scaled fleets.

1import math2
3
4def replicas_needed(traffic_rps: float, capacity_rps: float) -> int:5    return math.ceil(traffic_rps / capacity_rps)6
7
8def gpu_hours(day_n: int, day_h: int, night_n: int, night_h: int) -> int:9    return day_n * day_h + night_n * night_h10
11
12day = replicas_needed(50.0, 20.0)13night = replicas_needed(12.0, 20.0)14flat = 4 * 2415scaled = gpu_hours(day, 8, night, 16)16
17print(f"{day} {night}")  # -> 3 118print(f"{flat} {scaled}")  # -> 96 4019print(f"{flat - scaled}")  # -> 56

Three by day and 1 by night matches the hand count. Flat costs 96 GPU hours, scaled costs 40, and the gap is 56. Change night traffic to 25 and the first function returns 2, which is the signal to recheck headroom before trusting the saving.

Watch Out For

Packing GPUs to 100 percent and breaching p95

Use climbs to 98 percent and the dashboard looks efficient. Then p95 jumps from 180 ms to 420 ms because queue delay grows fast near saturation.

Symptom: p95 breaches only at peaks while the average looks fine, and larger batches make it worse. Fix: set the latency objective first, scale on queue depth or p95 rather than use alone, and keep one replica of headroom. Efficiency without a latency bound is overload.

Running live serving on interruptible capacity alone

Spot prices look far cheaper, so the whole serving fleet moves to spot. The provider reclaims capacity at peak, replicas vanish, and live traffic has nowhere to go.

Symptom: sudden drops with reclaim notices minutes before errors spike, always at the worst hour. Fix: keep live serving on on-demand or committed capacity, reserve spot for batch scoring and checkpointed training, and force a reclaim drill weekly.

The Quick Version

Serving bills come from idle GPUs, not slow models, so the platform pays only for capacity in use.
One replica holds 20 requests per second at p95 180 ms, so 50 needs 3 and 12 needs 1 by ceiling division.
Three replicas for 8 hours plus 1 for 16 hours costs 40 GPU hours against 96 flat, saving 56.
Right-size the instance first, scale on queue or p95 with headroom, and scale batch work to zero.
Keep live serving on on-demand capacity and push spot to batch and retraining that resumes from a checkpoint.
If p95 rises past its objective, the change was a slowdown, not a saving.