Core ML

Cost Per Token Engineering

You built an AI app that generates poems. It goes viral! You have 100,000 users. Then you get your AWS bill, and you owe $50,000. If it costs you $0.10 to generate a poem, but you only charge users $0.05, going viral will bankrupt you.

In traditional software, hosting costs are near zero. In GenAI, every single token generated burns a measurable fraction of a cent. If the Cost to Serve exceeds the Revenue per User, the business will fail.

Why Does This Exist?

In traditional software (like a Todo List app), the database and web server are so incredibly cheap that serving 1,000 users costs almost exactly the same as serving 10 users. The marginal cost of a new user is effectively zero.

In Generative AI, this is violently untrue. Every time a user clicks "Generate", the massive GPU does trillions of math operations. Running an H100 GPU costs about $\$ 3.00 $to$ $6.00 $an hour. If your model takes 10 seconds to generate a response, that single click just cost you exactly$ $0.01$ in electricity and hardware rental.

If a user clicks 100 times a day, they cost you $\$ 1.00 $a day, or$ $30 $a month. If you are only charging them$ $10 $a month for a subscription, **your company is losing$ $20$ every month on that user**.

Cost Per Token Engineering is the practice of optimizing your ML infrastructure to ensure that your unit economics are actually profitable.

Think of It Like This

Imagine you own a bakery that sells cupcakes for $\$ 3.00$.

If the flour, sugar, and electricity required to bake a single cupcake costs $\$ 4.00 $, you are losing$ $1.00 $on every sale. It doesn't matter how good your marketing is. In fact, if your marketing goes viral and you sell 10,000 cupcakes, you will instantly go bankrupt. You must "engineer" your recipe (buy cheaper flour, bake 50 cupcakes in the oven at once) to get the cost below$ $3.00$.

How to Calculate It

For LLMs, the standard industry metric is Cost per 1 Million Tokens. Here is how you calculate it for a self-hosted model:

Find your Hourly Server Cost: You rent an AWS server with 8x A100 GPUs for $\$ 32.00$ an hour.
Find your Throughput: You run a load test and discover that using dynamic-batching, your server can generate 5,000 tokens per second.
Calculate Tokens per Hour: $5,000 \times 3,600 = 18,000,000$ tokens per hour.
Calculate Cost per 1M Tokens: $\$ 32.00 / 18 = $1.77$ per 1 Million Tokens.

Now you know your baseline! If OpenAI charges $\$ 2.00 $per 1M tokens for GPT-4, you are slightly cheaper. If OpenAI charges$ $0.50$ per 1M tokens for GPT-3.5, you are vastly more expensive and should just use their API instead of hosting it yourself.

The Three Levers of Optimization

If your Cost Per Token is too high, you have three engineering levers to pull to lower it:

1. Batching (Increase Throughput)

As discussed in batch-vs-realtime-inference, processing one request at a time wastes GPU compute. By increasing your Batch Size from 1 to 32, you generate 32x more tokens per second using the exact same $\$ 32.00/hr server. This drastically lowers your Cost Per Token. (Tradeoff: It eats into your latency-budgets, making users wait longer).

2. Quantization (Decrease Hardware)

As discussed in model-quantization, compressing your model from 16-bit to 4-bit means you can fit it on a much cheaper GPU. Instead of paying $\$ 32.00/hr $for 8 GPUs, you can pay$ $4.00/hr$ for 1 GPU. (Tradeoff: It slightly degrades the intelligence of the model).

3. Smaller Models (Decrease Math)

Does your app really need a massive 70-Billion parameter model? If you fine-tune a tiny 8-Billion parameter model on your specific dataset, it requires 10x less math to run. It runs faster, fits on cheaper hardware, and plummets your Cost Per Token.

Watch Out For

The Input Token Trap. Many engineers only calculate the cost of the generated (output) tokens. But the model also has to process the user's prompt (the input tokens). If your app requires injecting a massive 50-page PDF into the prompt before the user can ask a question, the model has to process 30,000 input tokens every single time the user asks a question. Input tokens are cheaper to process than output tokens, but at high volumes, they will quietly destroy your unit economics. Use RAG (Retrieval-Augmented Generation) to only inject the 1 relevant page, not all 50.

The Quick Version

Cost Per Token Engineering ensures your ML product is actually profitable.
The industry standard metric is the Cost per 1 Million Tokens.
You calculate it by dividing your hourly server rental cost by the total number of tokens it can generate in an hour.
You can lower the cost by increasing throughput (Batching), shrinking the model footprint (Quantization), or simply using a smaller model.
You must carefully balance your desire for low costs against your strict Latency Budgets.