Inference is the second phase in a generative AI model’s lifecycle:
- Training: The process of learning model weights from data.
- Inference: Serving generative AI models in production.
In last decade’s machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle, both training and inference, for ML models.
Inference for classic ML models is relatively straightforward. In the early days of Baseten, we ran inference for models built with tools like XGBoost on lightweight CPUs with a simple software stack.
In contrast, inference for generative AI models is complex. You can’t simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. Doing inference well requires three layers:
- Runtime: Optimizing the performance of a single model on a single GPU-backed instance.
- Infrastructure: Scaling across clusters, regions, and clouds without creating silos while maintaining excellent uptime.
- Tooling: Providing engineers working on inference with the right level of abstraction to balance control with productivity.
These three layers must work together to create a system that can handle mission-critical inference at scale.
The runtime layer is responsible for ensuring that an individual model running on a GPU (or across several GPUs in a single instance) is running as performantly and efficiently as possible.
This layer depends on a sophisticated software stack, from CUDA to PyTorch to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains.
The runtime layer relies on a number of model performance techniques that apply new research to the unique challenges of inference on generative AI models:
- Batching: Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.
- Caching: Re-use the KV cache – the cached results of the attention algorithm – between requests that share prefixes.
- Quantization: Lower the precision of select pieces of the model to access more compute and reduce memory burden.
- Speculation: Generate and validate draft tokens to produce more than one token per forward pass during decode.
- Parallelism: Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks.
- Disaggregation: Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers.
These model performance techniques are used for models of all modalities, not just LLMs. Modalities like vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation extend the capabilities of AI systems and require their own inference optimizations.
These runtime optimizations are not enough. No matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle.
This is not a CUDA problem or a PyTorch problem. This is a systems problem that needs to be solved at the infrastructure layer.
The nature of infrastructure problems changes with each level of scale. At first, the problems are around autoscaling: knowing when to add and remove replicas and figuring out how to do so quickly.
Past a certain scale, generally a few hundred GPUs, infrastructure problems are defined by capacity. To get access to enough GPUs, inference engineers begin spreading workloads across multiple regions and multiple cloud providers.
This quickly leads to silos, where models in one cluster may be starved for resources while other clusters have unused capacity. The final level of scale in infrastructure is a global system that treats all available resources as a single unified pool of compute.
Thoughtful multi-cloud infrastructure also improves reliability, protecting against downtime in any individual region or cloud provider. And for global applications, running inference near end users improves end-to-end latency.
Once these runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Both inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide as the critical third layer in a complete inference platform.
Developer experience is subjective. For inference, one extreme is the black box: give a platform model weights, get back an API. At the other extreme is providing only basic constructs for compute, network, disk, and so forth.
The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, but enough abstraction to work productively. Inference Engineering presents a map of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling.
Chapter 1, Prerequisites, covers the product thinking and AI engineering work that need to be done before inference engineering comes into play: use case definition, latency and cost budgeting, and selecting and evaluating which generative AI models to optimize and deploy.
Chapter 2, Models, introduces the technical architecture of AI models – from large language models to image and video generation models – and establishes where the bottlenecks exist for inference with a special focus on optimizing attention.
Chapter 3, Hardware, starts at the spec sheet for modern GPUs and breaks down compute and memory, then disambiguates architectures and SKUs within NVIDIA’s datacenter-grade offerings before briefly surveying other accelerators on the market.
Chapter 4, Software, builds abstractions from CUDA to frameworks like PyTorch, Transformers, and Diffusers and inference engines like vLLM, SGLang, and TensorRT-LLM. It also introduces Dynamo, NVIDIA’s latest system for large-scale distributed model serving.
Chapter 5, Techniques, discusses key model performance optimization techniques adapted from cutting-edge research and applies them in production: quantization, speculative decoding, KV cache re-use, model parallelism, and disaggregation.
Chapter 6, Modalities, expands inference engineering beyond LLMs to voice and visuals. Many types of generative AI models – vision-language models, embedding models, automatic speech recognition (ASR) models, and speech synthesis models – adapt LLM architectures, meaning inference engineers can run them with the same tools and techniques used with LLMs. Image and video generation models have their own architectures and associated performance optimization techniques.
Chapter 7, Production, concludes the book with a rundown of the important problems to solve in operating infrastructure for and building performant applications on optimized model inference services.
Appendices A and B add a glossary of inference engineering terms and a collection of recommended resources for further reading, respectively.
Like LLMs, books have knowledge cutoffs. This book was finished in January 2026. While details will change, the principles, concepts, and foundational technologies in this book provide a strong background on inference engineering that will serve you well for years to come.