Inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, more reliable serving of generative AI models in production.
While the potential and impact of inference are becoming clear, the space is young. There are relatively few people working on inference, and newcomers can become experts quickly.
There are opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.
Inference Engineering is your guide to becoming an expert in inference. It contains everything that I’ve learned in four years of working at Baseten. This book is based on the hundreds of thousands of words of documentation, blogs, and talks I've written on inference; interviews with dozens of experts from our engineering team; and countless conversations with customers and builders around the world.
Inference Engineering presents a map of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling.
Chapter 1, Prerequisites, covers the product thinking and AI engineering work that need to be done before inference engineering comes into play: use case definition, latency and cost budgeting, and selecting and evaluating which generative AI models to optimize and deploy.
Chapter 2, Models, introduces the technical architecture of AI models – from large language models to image and video generation models – and establishes where the bottlenecks exist for inference with a special focus on optimizing attention.
Chapter 3, Hardware, starts at the spec sheet for modern GPUs and breaks down compute and memory, then disambiguates architectures and SKUs within NVIDIA’s datacenter-grade offerings before briefly surveying other accelerators on the market.
Chapter 4, Software, builds abstractions from CUDA to frameworks like PyTorch, Transformers, and Diffusers and inference engines like vLLM, SGLang, and TensorRT-LLM. It also introduces Dynamo, NVIDIA’s latest system for large-scale distributed model serving.
Chapter 5, Techniques, discusses key model performance optimization techniques adapted from cutting-edge research and applies them in production: quantization, speculative decoding, KV cache re-use, model parallelism, and disaggregation.
Chapter 6, Modalities, expands inference engineering beyond LLMs to voice and visuals. Many types of generative AI models – vision-language models, embedding models, automatic speech recognition (ASR) models, and speech synthesis models – adapt LLM architectures, meaning inference engineers can run them with the same tools and techniques used with LLMs. Image and video generation models have their own architectures and associated performance optimization techniques.
Chapter 7, Production, concludes the book with a rundown of the important problems to solve in operating infrastructure for and building performant applications on optimized model inference services.
Appendices A and B add a glossary of inference engineering terms and a collection of recommended resources for further reading, respectively.
1.1 Scale and Specialization
1.2 About Your App
1.2.1 AI-Native Applications
1.2.2 Online versus Offline
1.2.3 Consumer versus B2B
1.3 Model Selection
1.3.1 Model Evaluation
1.3.2 Fine-Tuning for Domain-Specific Quality
1.3.3 Distillation
1.4 Measuring Latency and Throughput
1.4.1 Latency Percentiles
1.4.2 End-to-End Metrics
2.1 Neural Networks
2.1.1 Linear Layers and Matmul
2.1.2 Activation Functions
2.2 LLM Inference Mechanics
2.2.1 LLM Architecture
2.2.2 Transformer Blocks
2.2.3 Attention
2.2.4 Mixture of Experts Models
2.3 Image Generation Inference Mechanics
2.3.1 Image Generation Model Architecture
2.3.2 Few-Step Image Generation Models
2.3.3 Video Generation
2.4 Calculating Inference Bottlenecks
2.4.1 Ops:Byte Ratio and Arithmetic Intensity
2.4.2 LLM Inference Bottlenecks
2.4.3 Image Generation Inference Bottlenecks
2.5 Optimizing Attention
3.1 GPU Architecture
3.1.1 Compute
3.1.2 Memory and Caches
3.2 GPU Architecture Generations
3.2.1 Hopper GPUs
3.2.2 Ada Lovelace GPUs
3.2.3 Blackwell GPUs
3.2.4 Rubin GPUs
3.2.5 Grace and Vera CPUs
3.3 Instances
3.3.1 Multi-GPU Instances
3.3.2 Multi-Instance GPUs
3.4 Other Datacenter Accelerator Options
3.5 Local Inference
3.5.1 Desktop Inference
3.5.2 Mobile Inference
4.1 CUDA
4.1.1 CUDA Kernels for Inference
4.1.2 CUDA Kernel Selection
4.1.3 Reducing Memory Accesses with Kernel Fusion
4.2 Deep Learning Frameworks and Libraries
4.2.1 PyTorch
4.2.2 Model File Formats
4.2.3 ONNX Runtime and TensorRT
4.2.4 Transformers and Diffusers
4.3 Inference Engines
4.3.1 vLLM
4.3.2 SGLang
4.3.3 TensorRT-LLM
4.4 NVIDIA Dynamo
4.5 Performance Benchmarking and Load Testing
4.5.1 Performance Benchmarking Tooling
4.5.2 Performance Benchmarking Tips
4.5.3 Profiling Performance
5.1 Quantization
5.1.1 Number Formats
5.1.2 Quantization Approaches
5.1.3 Measuring Quality Impact
5.2 Speculative Decoding
5.2.1 Draft-Target Speculative Decoding
5.2.2 Medusa
5.2.3 EAGLE
5.2.4 N-gram Speculation and Lookahead Decoding
5.3 Caching
5.3.1 Prefix Caching and KV Cache Re-Use
5.3.2 Where to Store the KV Cache
5.3.3 Cache-Aware Routing
5.3.4 Long Context Handling
5.4 Model Parallelism
5.4.1 Tensor Parallelism for Lower Latency
5.4.2 Expert Parallelism for Higher Throughput
5.4.3 Multi-Node Inference
5.5 Disaggregation
5.5.1 How Disaggregation Works
5.5.2 When to Use Disaggregation
5.5.3 Dynamic Disaggregation with NVIDIA Dynamo
6.1 Vision Language Models
6.1.1 Video Processing for Vision Language Models
6.1.2 Omni-Modal Models
6.2 Embedding Models
6.2.1 Embedding Model Architecture
6.2.2 Embedding Model Inference
6.3 ASR Models
6.3.1 Single-Chunk Latency Optimization
6.3.2 Long File Latency Optimization
6.3.3 Diarization
6.4 TTS Models
6.4.1 Streaming Real-Time Text to Speech
6.4.2 Speech-to-Speech Models
6.5 Image Generation Models
6.5.1 Image Generation Kernel Optimization
6.5.2 One Weird Trick for Faster Image Generation
6.6 Video Generation Models
6.6.1 Attention Optimization and Quantization
6.6.2 Context Parallelism
7.1 Containerization
7.1.1 Dependency Management
7.1.2 NIMs
7.2 Autoscaling
7.2.1 Concurrency and Batch Sizing
7.2.2 Cold Starts
7.2.3 Routing, Load Balancing, and Queueing
7.2.4 Scale to Zero
7.2.5 Independent Component Scaling
7.3 Multi-Cloud Capacity Management
7.3.1 GPU Procurement
7.3.2 Geo-Aware Load Balancing
7.3.3 Building for Reliability
7.3.4 Security and Compliance
7.4 Testing and Deployment
7.4.1 Zero-Downtime Deployment
7.4.2 Cost Estimation
7.4.3 Observability
7.5 Client Code
7.5.1 Client Latency Overhead
7.5.2 Asynchronous Inference
7.5.3 Streaming and Protocol Support
7.6 Production Inference with Baseten
Philip Kiely joined Baseten in January 2022 and is a software developer and author. He has spoken on inference at conferences like NVIDIA GTC, PyTorch Conference, AI Engineer World’s Fair, and AWS re:Invent. Outside of work, you'll find Philip practicing martial arts, reading a new book, or cheering for his adopted SF sports teams.


Inference is the process of running an AI model like an LLM or image generation model in production to generate output in response to user queries. Inference powers every AI application and is the most important category in the space.
There will be a million inference engineers in a couple of years, with AI labs, AI infrastructure companies, startups, and enterprises alike presenting insatiable demand for inference on open source, fine-tuned, and custom-build AI models.
Today, there are relatively few engineers working on inference, and there are novel, important, and deeply technical problems to solve at every layer of the stack. You have an opportunity to be early on the most important technology of our generation.
I tried my hardest to use AI as an assistant for everything from research to proofreading. While there were a few places it was able to accelerate my work, like appendix alphabetization and fixing inconsistencies in figure numbering, I had to write this book myself and work with talented human designers for the cover and interior graphics to reach the quality bar I was aiming for.
Inference Engineering is written for technically-minded people of all backgrounds who want to deepen their understanding of inference. Anyone can read the first twenty pages, but after that a basic familiarity with coding and computer science concepts will be helpful.
There are dozens of technologies that work together to make inference possible. This book covers a wide range of technologies up and down the stack, though there is a definite focus on NVIDIA GPUs and inference in the datacenter.
Inference Engineering is 256 pages, which is about 46,000 words or about 60,000 tokens. I recommend against trying to feed it to your agent in a single chunk.
Inference Engineering is available in PDF, EPUB, paperback, hardcover, and audiobook (M4B) formats.
Yes! I worked with Rime to create a custom voice clone of myself and built a multi-stage pipeline to generate an audio edition of Inference Engineering. Your free digital download includes the audiobook.
You can download a free PDF from this website, and you can find me and Baseten at events around SF as well as conferences like NVIDIA GTC and AWS re:Invent to get a free paperback copy. Physical and digital copies are also available for sale on Amazon.