Chapter 2: Models

2.4 Calculating Inference Bottlenecks

Using ops:byte ratio and arithmetic intensity to determine whether a workload is compute bound or memory bound.

In a perfectly optimized system, every resource is fully utilized at all times. In GPUs, there are two main resources:

  • Compute: The number of floating-point operations per second that the GPU can achieve.
  • Memory bandwidth: The number of bytes that the GPU can move per second.

Ideally, compute is never sitting idle waiting for information from memory, and memory bandwidth never goes unused waiting for compute to finish.

In the real world, systems have bottlenecks: imbalances where one resource is idle while another is saturated. Discovering these bottlenecks is the first step to improving performance. If a certain operation is bottlenecked on memory bandwidth, no amount of compute optimization will make the system faster, and vice versa.

In most cases, inference systems have the following bottlenecks:

  • LLM prefill (KV cache construction) is compute bound.
  • LLM decode (token generation) is memory bound.
  • Image and video generation are compute bound.

When optimizing performance on each of these phases, the goal is to make the bottleneck less limiting to system-wide performance. For example, batching multiple requests together makes LLM decode less memory bound because processing a batch of requests uses more compute for the same amount of memory traffic.

2.4.1 Ops:Byte Ratio and Arithmetic Intensity

Each GPU has a specific compute speed (measured in operations per second) and memory bandwidth (measured in gigabytes or terabytes per second). Compare these to determine the ops:byte ratio of a given GPU.

For example, an H100 GPU in FP16 can perform 989 teraFLOPS of dense computation against 3.35 TB/s of memory bandwidth. This yields an ops:byte ratio of about 295.

For inference in FP16 to be perfectly balanced (as all things should be) on an H100 GPU, the inference system needs to perform 295 floating point operations for every byte of memory it accesses.

To figure out that ratio, calculate the arithmetic intensity of the algorithm. Arithmetic intensity, also known as operational intensity, is the ratio between work and memory traffic for the calculation at hand.

Figure 2.12: The equation for arithmetic intensity.
Figure 2.12: The equation for arithmetic intensity.

Where ops:byte was measured on a per-second scale, arithmetic intensity is measured across the execution of a single function or algorithm.

Arithmetic intensity is visualized with a roofline model, which charts performance against the bandwidth ceiling (a diagonal line) and the performance ceiling (a horizontal line).

Figure 2.13: A roofline chart shows the switch from memory to compute bottleneck based on arithmetic intensity.
Figure 2.13: A roofline chart shows the switch from memory to compute bottleneck based on arithmetic intensity.

Plotting against the roofline model reveals if the algorithm is:

  • Compute bound: When the arithmetic intensity is higher than the hardware’s ops:byte ratio and hits the horizontal performance ceiling, it’s compute bound.
  • Memory bound: When the arithmetic intensity is lower than the hardware’s ops:byte ratio and hits the diagonal bandwidth ceiling, it’s memory bound.

To find a bottleneck, look at arithmetic intensity for the most expensive calculations in a system. For inference, one such calculation is attention.

2.4.2 LLM Inference Bottlenecks

LLM inference has two phases:

  • Prefill: Determines the time to first token (TTFT) and is compute-bound.
  • Decode: Determines the tokens per second (TPS) and is memory-bound.

For each phase, you can prove the existence of the bottleneck by comparing the arithmetic intensity of the most important operation to the ops:byte ratio of available hardware.

In both prefill and decode, the most expensive operation is attention. The exact arithmetic intensity of attention depends on the model architecture (dimensions, heads, etc), the input sequence length, and the implementation of the attention algorithm.

The essential difference is that prefill processes the entire input sequence in parallel, while decode generates tokens one at a time.

For prefill, the model weights are loaded a single time, then a series of large matrix multiplication between the matrix of inputs and the attention matrices occurs. This is a lot of calculations versus a single read from memory, creating a high arithmetic intensity.

On decode, the model weights are loaded for every token, which is generated via much less-expensive vector-matrix multiplication. In this case, relatively few floating-point operations are needed compared to loading the entire model weights, so the arithmetic intensity is low.

As an example of calculating exact arithmetic intensity, consider a decode step for a model with a 128-dimensional attention head (d=128) on a sequence of 4096 tokens (N=4096). For this analysis, use the standard algorithm for attention without any optimizations.

Figure 2.14: Standard attention implementation, adapted from “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness” (Dao et al., 2022).
Figure 2.14: Standard attention implementation, adapted from “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness” (Dao et al., 2022).

Based on the parameters of this exercise, establish the size of these matrices:

  • N: The sequence length, established as 4096.
  • d: The dimensionality of the attention head, set to 128.
  • Q, K, V: Given as Nxd, or 4096x128.
  • S, P: Calculated as NxN, or 4096x4096.
  • O: Calculated as Nxd, or 4096x128

Assume FP16 inference, where each value in the matrix is two bytes. For reference, a 4096x4096 matrix is about 32 MiB, or about the same amount of data as a high-resolution RAW DSLR photo.

Each of the three lines of the attention algorithm follows the same pattern: load data from memory, perform a calculation, and store the result to memory.

Figure 2.15: Memory movement (reads and writes) and compute work for the attention implementation in Figure 2.14.
Figure 2.15: Memory movement (reads and writes) and compute work for the attention implementation in Figure 2.14.

To calculate the total memory movement, sum the first and third columns, which track reads from and writes to GPU memory:

Figure 2.16: The total memory movement for a kernel is the sum of all reads and writes across the three steps.
Figure 2.16: The total memory movement for a kernel is the sum of all reads and writes across the three steps.

To calculate the total compute, sum the second column:

Figure 2.17: The total compute for the kernel is the sum of operations across the three steps.
Figure 2.17: The total compute for the kernel is the sum of operations across the three steps.

To calculate the arithmetic intensity, compare the work (total compute) to the memory traffic:

Figure 2.18: The arithmetic intensity of a kernel is the total work (number of compute operations) divided by the memory movement
Figure 2.18: The arithmetic intensity of a kernel is the total work (number of compute operations) divided by the memory movement

For this example, the arithmetic intensity of 62 is much lower than the H100 GPU’s ops:byte ratio of 295. The exact numbers vary by model, sequence length, and hardware, but this example illustrates the general principle that decode is memory bound.

Calculating arithmetic intensity like this is an academic exercise, not a routine task for inference engineers. But seeing it once is useful for building intuition.

2.4.3 Image Generation Inference Bottlenecks

Image and video models are relatively small – they have a tenth as many parameters as frontier language models – but their attention mechanism is just as computationally demanding.

Image and video generation models use iterative denoising, not autoregressive token generation.

Just like attention for LLM prefill processes the entire input sequence at once, attention for generating media must consider the entire image or video object as represented in latent space.

Also like LLM prefill, image and video generation model inference is bottlenecked on compute. Specific techniques for optimizing inference for these modalities are featured in sections 6.5 and 6.6.