Inference Engineering

Chapter 3: Hardware

GPU architecture, NVIDIA generations from Hopper to Rubin, multi-GPU instances, alternative accelerators, and local inference.

Inference engineering relies on accelerators: powerful hardware designed to load terabytes of data and perform trillions of operations per second.

The most common type of accelerator for inference is the GPU, and the market leader in GPUs for inference is NVIDIA. This book focuses on inference engineering for NVIDIA GPUs in the datacenter, but section 3.4 of this chapter covers other vendors of datacenter accelerators, and section 3.5 covers local inference.

Across vendors, there are three types of GPUs on the market:

  • Datacenter GPUs: Racked servers with interconnected high-performance GPUs. Example: NVIDIA B200.
  • Workstation GPUs: Individual desktop GPUs for professional workflows. Example: NVIDIA RTX Pro 6000.
  • Personal computing GPUs: Individual desktop GPUs for everyday use. Example: NVIDIA GeForce RTX 5090.

Inference at scale uses datacenter GPUs mounted on racks: refrigerator-sized chassis with standardized power, networking, and cooling.

Datacenter GPUs like the NVIDIA B200 offer the highest individual performance, but more importantly, include high-bandwidth GPU-to-GPU interconnects, are installed in highly standardized configurations, and are available by the millions in datacenters worldwide.

I doubt that you have a B200 GPU running under your desk. If you do, send me a picture! Instead, inference on datacenter GPUs runs in one of three modes:

  • Cloud: GPUs are rented in someone else’s datacenter, usually hyperscalers like AWS and GCP or neoclouds like Coreweave and Nebius.
  • On-premise: GPUs are purchased and installed in a datacenter that you control directly.
  • Air-gapped: GPUs are installed on-premise and you need to physically access the GPUs to run inference.

Most inference engineers use cloud GPUs. Large enterprises and governments run on-premise and air-gapped deployments, but cloud-based GPUs offer the flexibility and access that fast-growing AI products need to scale.

Even with these constraints, navigating the hardware landscape is complex. From variations among cloud providers to NVIDIA’s own naming conventions, there are many nuances in selecting the right accelerator.

  • 3.1 GPU Architecture: Streaming multiprocessors, tensor cores, and the memory hierarchy from HBM through caches to registers.
  • 3.2 GPU Architecture Generations: What changed across Hopper, Ada Lovelace, Blackwell, and Rubin, plus the Grace and Vera CPUs paired with them.
  • 3.3 Instances: Multi-GPU instances, NVLink and InfiniBand interconnect, and slicing a card with multi-instance GPU.
  • 3.4 Other Datacenter Accelerator Options: AMD GPUs, TPUs, and custom inference silicon as alternatives to NVIDIA hardware.
  • 3.5 Local Inference: Running models on desktops and phones, and the constraints that separate edge inference from the datacenter.