Inference Engineering

Chapter 4: Software

The inference software stack, from CUDA kernels up through PyTorch, vLLM, SGLang, TensorRT-LLM, and Dynamo.

NVIDIA’s market dominance in the inference space is in no small part due to the robust and mature software ecosystem around its hardware.

Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most yearly, with two-year release cycles being more common.

But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other prerelease version of each of your software dependencies just to get support for the new model.

Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. While hardware centers on NVIDIA and a few competitors, there are countless companies building software at various levels of the inference stack.

For inference engineers, these are some key players:

  • NVIDIA: Invests heavily in its own sometimes-proprietary software ecosystem, from CUDA up to Dynamo.
  • Hugging Face: Maintains a model registry for all open models plus transformers and diffusers.
  • The Linux Foundation: Maintains hardware-agnostic projects like PyTorch and vLLM.
  • LMSYS Org: Develops essential tools for inference and evaluation, most notably SGLang.

There are thousands more companies, universities, and research institutions making essential open-source contributions to inference.

The software space is too big and too fast to exhaustively document in any book, much less in a single chapter. Instead, this chapter presents foundational technologies with long-term relevance.

Throughout the chapter, technologies are presented with increasing levels of abstraction:

  • CUDA: Direct communication to the GPU for explicit control over computations and memory (section 4.1).
  • Deep learning frameworks: Abstractions over CUDA for training, exporting, and running neural networks in Python (section 4.2).
  • Inference engines: Highly configurable PyTorch-backed inference for common architectures (section 4.3).
  • NVIDIA Dynamo: Sits on top of inference engines to power large-scale deployments (section 4.4).

Most inference engineering today happens at the higher levels of abstraction, configuring and deploying inference engines and orchestrating inference across multiple GPUs. No matter what level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work.