Core ML

Compilation and Kernel Fusion

Running 10 small math equations requires the GPU to look at the instructions 10 separate times. Kernel fusion mathematically combines those 10 small equations into 1 massive equation, allowing the GPU to do it all at once.

In a standard PyTorch execution, each mathematical operation (Multiply, Add, ReLU) requires the GPU to read from and write back to VRAM. Kernel Fusion compiles these into a single CUDA operation, skipping the intermediate VRAM trips.

Why Does This Exist?

To understand Kernel Fusion, you must understand the biggest bottleneck in machine learning: Memory Bandwidth.

Modern GPUs (like the H100) are so incredibly fast at doing math that they spend most of their time simply waiting for data to travel from the VRAM memory chips into the actual calculation cores. If your model requires you to Multiply, then Add, and then apply ReLU, a standard execution looks like this:

Load data from VRAM. Compute Multiply. Write intermediate result back to VRAM.
Load intermediate result from VRAM. Compute Add. Write intermediate result back to VRAM.
Load intermediate result from VRAM. Compute ReLU. Write final result to VRAM.

This requires 3 trips to VRAM. Because VRAM transfers are relatively slow, the GPU is severely bottlenecked.

Kernel Fusion takes those three separate operations and writes a single, custom C++ (CUDA) program that does ReLU(Add(Multiply)). Now, the GPU loads the data from VRAM exactly once, does all three math operations in the fast computation cores, and writes the final result back to VRAM exactly once. You just made your model 3x faster without changing the math.

Think of It Like This

Imagine an assembly line building sandwiches.

Standard Execution: Worker 1 puts bread on a plate, walks the plate to a refrigerator, and puts it inside. Worker 2 takes the plate out of the fridge, adds ham, and puts it back in the fridge. Worker 3 takes it out, adds cheese, and puts it back. The workers spend 90% of their time just walking to the fridge.

Kernel Fusion: You combine the three jobs into one station. A single worker grabs the bread, adds ham, and adds cheese all at once, and then puts the finished sandwich in the fridge exactly once.

How It Works: Model Compilation

You do not write fused CUDA kernels by hand (unless you are a researcher inventing a new architecture, like the creators of FlashAttention). Instead, you use a Compiler (like torch.compile or TensorRT).

When you start your production server, the Compiler spends a few minutes analyzing your model's mathematical graph. It looks for common patterns (like a matrix multiplication followed immediately by an activation function). When it finds them, it automatically generates a highly optimized, fused CUDA kernel specifically for that exact sequence of operations.

Graph Capture

Compilers also perform Graph Capture (e.g., CUDA Graphs). In standard Python, every single time your model processes a request, the Python CPU code has to issue instructions to the GPU. This CPU-to-GPU communication takes a few milliseconds. Graph Capture records the exact sequence of GPU instructions on the first run. For all future runs, it bypasses Python entirely and just tells the GPU to replay the recorded graph.

Show Me the Code

In modern PyTorch 2.0, compiling a model for production requires literally one single line of code.

1import torch2import torchvision.models as models3
4# 1. Load standard eager PyTorch model5model = models.resnet50()6
7# 2. Compile it! 8# Under the hood, PyTorch uses OpenAI's Triton compiler to 9# analyze the graph, fuse the kernels, and generate optimized C++ code.10compiled_model = torch.compile(model)11
12# 3. The first run will be very slow (it is compiling the C++ code)13dummy_input = torch.randn(1, 3, 224, 224)14compiled_model(dummy_input)15
16# 4. All subsequent runs will be drastically faster!17# Ready for production serving.

Watch Out For

The Cold Start Problem. Because compilation requires generating, compiling, and testing custom C++ code for your specific hardware, the very first time you run a compiled model, it might take several minutes before it returns a prediction. If you deploy this to a web server that scales up and down automatically (Serverless Autoscaling), every time a new server spins up, the first user to hit it will be forced to wait 5 minutes while the model compiles. You must pre-compile your models (Ahead-of-Time compilation) or use Warm-Up scripts before opening the server to internet traffic.

The Quick Version

GPUs are so fast that they spend most of their time waiting for data to load from memory (the Memory Bandwidth bottleneck).
Standard execution writes intermediate results to memory after every single math operation.
Kernel Fusion combines multiple math operations into a single CUDA program, meaning data is loaded from memory exactly once.
You achieve this using Model Compilers (like torch.compile, TensorRT, or XLA), which analyze your model and automatically write the fused C++ code.
Compilation happens on the first run, leading to a massive delay (Cold Start) before the model achieves its incredibly fast steady-state speed.

Related concepts

GPU Utilization and Profiling