Core ML

Training Cost Estimation

Before you spin up 1,000 GPUs, you need to know if the bill will be $50,000 or $5,000,000. Cost estimation is a math formula that predicts the financial budget based on the model's size, the dataset's size, and the hardware's theoretical speed.

Estimating cost requires knowing the total FLOPs required (6 * Parameters * Tokens), the theoretical speed of the GPU, the actual utilization efficiency (MFU), and the hourly rental cost of the hardware.

Why Does This Exist?

In traditional software engineering, compute costs are usually an afterthought. A web server costs $\$ 40 $a month, and if you write inefficient code, maybe it costs$ $80$.

In Large Language Model training, compute is the primary capital expenditure. Training Llama 3 70B cost tens of millions of dollars. If a research scientist makes a configuration error that slows down training by 15%, that error literally costs millions of dollars in wasted GPU rental time.

You cannot simply "start a training run and see how much it costs." You must calculate the exact budget required to train the model to convergence before you rent the hardware. Training Cost Estimation is the mathematical framework used to predict that budget.

Think of It Like This

Imagine you need to dig a 100-mile trench. You know a shovel scoops 5 pounds of dirt per minute. You know a backhoe scoops 500 pounds of dirt per minute, but costs $\$ 200$ an hour to rent.

To estimate the cost of the project, you don't just start digging. You calculate the total pounds of dirt in a 100-mile trench, divide it by the backhoe's scooping speed, multiply by the hourly rental rate, and add a 20% margin for when the backhoe inevitably breaks down. That is exactly how we estimate GPU training costs.

The Math: From FLOPs to Dollars

1. The Total Work (Total FLOPs)

A FLOP is a Floating Point Operation (e.g., adding or multiplying two decimal numbers). To calculate the total number of math operations required to train a Transformer, we use a standard approximation rule: It takes exactly 6 FLOPs per parameter per token. (2 for the forward pass, and 4 for the backward pass).

$\text{Total FLOPs} = 6 \times \text{Number of Parameters} \times \text{Number of Tokens}$

Example: A 70-Billion parameter model trained on 1 Trillion tokens. $6 \times (70 \times 10^9) \times (1 \times 10^{12}) = 4.2 \times 10^{23} \text{ FLOPs}$

2. Hardware Speed and MFU

You plan to rent an Nvidia H100 GPU. The datasheet says an H100 can process $989 \times 10^{12}$ FLOPs per second (TFLOPS) in bfloat16. However, you will never hit this number. The GPUs spend time communicating over the network, waiting for the CPU, or stalling on memory bandwidth.

The percentage of the theoretical maximum speed you actually achieve is called the Model FLOPs Utilization (MFU). A world-class training run achieves an MFU of about $50\%$ . A poorly optimized run might hit $20\%$ .

$\text{Real Speed} = \text{Theoretical TFLOPS} \times \text{MFU}$ Example: $989 \text{ TFLOPS} \times 0.50 = 494.5 \text{ TFLOPS per GPU}$

3. Calculating the Time and Cost

Now that you know the total work, and the real speed of your shovel, you can calculate the time.

$\text{Total Seconds} = \frac{\text{Total FLOPs}}{\text{Real Speed per GPU} \times \text{Number of GPUs}}$

Let's assume you rent 1,000 H100 GPUs at $\$ 3.00$ per hour per GPU.

Total FLOPs: $4.2 \times 10^{23}$
Cluster Real Speed: $1000 \times (494.5 \times 10^{12}) = 4.945 \times 10^{17}$ FLOPs per second.
Total Seconds: $4.2 \times 10^{23} / 4.945 \times 10^{17} = 849,342$ seconds.
Total Hours: $849,342 / 3600 = 235.9$ hours.
Total Cost: $235.9 \text{ hours} \times 1,000 \text{ GPUs} \times \$ 3.00 = $707,700$.

Watch Out For

Ignoring the Uptime Factor. The formula above assumes your 1,000 GPUs run perfectly for 235 hours straight. As we learned in fault-tolerant-training, they won't. Hardware will fail, training will pause, and checkpoints will need to be reloaded. You must factor in Cluster Uptime (often between 85% and 95%). If your uptime is 90%, you must divide your total estimated time by 0.9, immediately adding 10% to your final financial budget.

The Quick Version

Training costs are estimated using a rigid mathematical formula before any GPUs are rented.
Total Work is roughly $6 \times \text{Parameters} \times \text{Tokens}$ .
MFU (Model FLOPs Utilization) is the percentage of theoretical GPU speed you actually achieve (usually 40-50%).
Real-world costs must always include a buffer for Uptime, accounting for hardware failures and cluster restarts.