# 3.1 GPU Architecture

_Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 3: Hardware. [Full book index](https://www.baseten.co/inference-engineering/llms.txt)

GPUs are throughput machines. Where CPUs are great at complex sequential execution, GPUs are designed for simple, massively parallel workloads.

Specifically, GPUs are great at performing one uniform operation on thousands of independent pieces of data. Given that AI model inference is a series of vector and matrix multiplications, GPUs are a natural fit.

While the principle of highly parallel computation is simple, GPUs themselves are extraordinarily complex pieces of technology. Hardware engineering is a fascinating multidisciplinary field, from the physics of powering and cooling the chips to the impossibly tight tolerances involved in manufacturing each component.

Inference engineers work at a comfortable level of abstraction above the GPU hardware, but a strong mental model for what’s going on inside the box is essential for building high-performance systems.

## 3.1.1 Compute

If you’re familiar with CPUs, you’ve probably heard about cores, like an 8-core Intel i9 CPU in a high-end gaming computer.

In GPUs, cores have a different meaning. GPUs have Streaming Multiprocessors (SMs), while each SM contains multiple cores. There are three types of compute in GPUs:

- **CUDA Core:** Operates on individual numbers (scalars).
- **Tensor Core:** Operates on vectors and matrices.
- **Special Function Unit (SFU):** Accelerates certain mathematical operations like sin, cos, and log.

When measuring GPU compute for inference, measure in terms of Tensor Core compute. SFUs are essential for softmax, but Tensor Cores are responsible for Matrix Multiply and Accumulate (MMA) instructions, which are foundational to inference.

The “accumulate” step in MMA means adding the product of two matrices to a base matrix to produce the output, as shown in Figure 3.1.

![Figure 3.1: Matrix Multiply and Accumulate (MMA) multiples matrix A by matrix B, adds matrix C, and stores the result as matrix D.](https://www.datocms-assets.com/104802/1788123295-inference-engineering-figure-3-1.png)
_Figure 3.1: Matrix Multiply and Accumulate (MMA) multiples matrix A by matrix B, adds matrix C, and stores the result as matrix D._

Unlike cores, the concept of a thread is similar between CPUs and GPUs. Where a CPU has dozens to hundreds of threads, GPUs have tens to hundreds of thousands of threads that can work concurrently, switch tasks in a single clock cycle, and execute simple instructions in parallel.

Compute is measured in FLOPS (floating point operations per second) and datacenter GPUs are capable of trillions or quadrillions of FLOPS (teraFLOPS and petaFLOPS, respectively). However, when you read a spec sheet, you’ll see two measurements for Tensor Core compute:

- **Dense:** The raw floating-point operations per second if every element of the tensor is used.
- **Sparse:** In tensors with 2:4 structured sparsity, where 50 percent of the values are 0, Tensor Cores can skip multiplication by 0.

A GPU’s FLOPS at a given precision with sparsity are often, but not always, double that of dense operations at the same precision. By default, inference is dense, so ensure that you’re looking at FLOPS without sparsity.

FLOPS generally double with each halving of precision. A GPU capable of one petaFLOPS on 16-bit numbers will be able to do two petaFLOPS on 8-bit numbers. This is relevant for inference – be sure to compare FLOPS across GPUs at identical precisions.

Compute is the bottleneck for LLM prefill and for image and video generation. If you’re selecting hardware with one of these use cases in mind, pick the accelerator with more FLOPS.

## 3.1.2 Memory and Caches

GPUs contain high-speed onboard memory called VRAM. Just like the “G” in GPU stands for graphics, the “V” in VRAM stands for video – a callback to these accelerators’ original purpose.

Today, VRAM is added to GPUs in the form of HBM3, HBM3e, or HBM4, all various generations of high-bandwidth memory. GPUs have dozens or hundreds of gigabytes of VRAM.

There are two types of memory on any chip, CPU or GPU:

- **DRAM (Dynamic RAM):** General-purpose off-chip memory denominated in gigabytes.
- **SRAM (Static RAM):** Faster, more expensive, on-chip memory denominated in kilobytes or megabytes.

VRAM is a type of DRAM. GPUs also feature SRAM on-chip in the form of caches. GPUs have three levels of cache:

- **L0:** Instruction cache for a single Tensor Core.
- **L1:** Shared memory per Streaming Multiprocessor.
- **L2:** Global cache across Streaming Multiprocessors.

An H100 GPU has 256 KB of L1 cache per Streaming Multiprocessor and 50 MB total L2 cache on chip.
VRAM bandwidth measures the peak transfer rate between GPU cores and VRAM via the memory bus. In practice, this determines how quickly VRAM can supply data into the GPU’s cache hierarchy.

![Figure 3.2: A GPU has multiple SMs, each of which has multiple Tensor Cores. L1 cache sits within SMs and L2 cache is shared.](https://www.datocms-assets.com/104802/1788123300-inference-engineering-figure-3-2.png)
_Figure 3.2: A GPU has multiple SMs, each of which has multiple Tensor Cores. L1 cache sits within SMs and L2 cache is shared._

The total amount of VRAM on a GPU limits the size of the model you can load onto it. The VRAM should hold the model weights, plus at least 50 percent headroom for KV cache (more for long context, high batch sizes, or video generation models).

If there isn’t enough VRAM available for the weights, loading the model will fail with an OOM (out of memory) error. And if there isn’t enough headroom, inference will be slow or crash with an OOM.

Memory bandwidth is the bottleneck for LLM decode at low to medium batch sizes. High-end GPUs have terabytes per second of memory bandwidth. If you are picking a GPU and want to generate more tokens per second, select the accelerator with a higher memory bandwidth, like the H200 instead of the H100.