GPU Utilization and Profiling

Why Does This Exist?

If you want to know how hard your computer is working, you open the Task Manager and look at CPU usage (e.g., 85%). For GPUs, engineers type nvidia-smi into the terminal. It might say Volatile GPU-Util: 100%. You might assume you have perfectly optimized your ML code. You haven't.

nvidia-smi is dangerously misleading. It only checks if any part of the GPU was active during the last second. Modern GPUs have distinct sections: Tensor Cores (the hyper-fast math calculators) and Memory Bandwidth (the pipes that move data). If your code is terribly unoptimized, the Tensor Cores might sit completely idle while the Memory pipes struggle to move data. nvidia-smi will still proudly report 100% utilization.

To actually optimize your code, you must use a Profiler. A profiler records exactly what every microscopic transistor of the GPU is doing at every single microsecond, generating a visual timeline (a Trace) so you can find the true bottlenecks.

Think of It Like This

The Two Types of Bottlenecks

When you read a Profiler Trace, you are looking to see which of these two bottlenecks is slowing down your model:

1. Memory-Bound

Your math is very fast, but loading the data is very slow.

• The Trace shows: Long, wide blocks of memory transfers, followed by tiny, microscopic slivers of actual math execution.
• The Fix: You need to move less data. Use compilation-and-kernel-fusion to merge operations, or use model-quantization to shrink the size of the data so it travels faster.

2. Compute-Bound

Your data loads instantly, but the math is incredibly complex and takes a long time.

• The Trace shows: Instant data transfers, followed by massive, thick blocks of math execution where the Tensor Cores are running at maximum capacity.
• The Fix: Congratulations! This is exactly what you want. The only way to go faster here is to buy a better GPU or redesign your neural network architecture.

Show Me the Code

PyTorch comes with a built-in Profiler that generates a JSON trace file. You can then open this file in Google Chrome by typing chrome://tracing in your URL bar.

1import torch2import torch.profiler3import torchvision.models as models4
5model = models.resnet18().cuda()6inputs = torch.randn(5, 3, 224, 224).cuda()7
8# Start the Profiler9with torch.profiler.profile(10    activities=[11        torch.profiler.ProfilerActivity.CPU,12        torch.profiler.ProfilerActivity.CUDA,13    ],14    # Save the visual timeline to a file15    on_trace_ready=torch.profiler.tensorboard_trace_handler('./log_dir')16) as prof:17    18    # Run the math19    model(inputs)20    21    # Tell the profiler this step is done22    prof.step()23
24# Now open Chrome -> chrome://tracing -> Load the JSON file

For absolute maximum precision, hardcore ML engineers skip the PyTorch profiler and use Nvidia's native tool, nsys (Nsight Systems), via the command line.

Watch Out For

The Quick Version

• nvidia-smi lies. 100% GPU utilization does not mean your code is running efficiently.
• You must use a Profiler (PyTorch Profiler or nsys) to generate a microsecond-level visual timeline of execution.
• If your trace shows long memory transfers and short math execution, your code is Memory-Bound.
• If your trace shows short memory transfers and long math execution, your code is Compute-Bound (which is the goal).
• Massive empty gaps in a GPU trace usually indicate that the Python CPU code is too slow to feed the GPU.

What to Read Next

• compilation-and-kernel-fusion — The primary software technique used to fix Memory-Bound bottlenecks.
• model-quantization — The primary mathematical technique used to fix Memory-Bound bottlenecks.