# Inference Engineering > Inference Engineering by Philip Kiely (Baseten Books, 2026) is a complete guide to serving generative AI models in production, covering model and GPU fundamentals, inference engines, optimization techniques, every major modality, and the infrastructure to run it all at scale. _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. The complete text is published here as Markdown. Every chapter is one file, and every numbered section is also its own file, so you can fetch just the part you need. Chapters run 7 to 49 KB; sections run 2 to 8 KB. Read the expanded table of contents first if you are not sure where to look. - [Expanded table of contents](https://www.baseten.co/inference-engineering/book/overview.md): Every chapter's opening paragraphs plus a one-line summary of all 42 sections. ## Start Here - [Preface](https://www.baseten.co/inference-engineering/book/preface.md): Why inference engineering emerged as a discipline, and how open models made it essential for AI product builders. - [Chapter 0: Inference](https://www.baseten.co/inference-engineering/book/00-inference.md): The three layers of an inference stack -- runtime, infrastructure, and tooling -- and why generative AI inference is hard. ## Chapter 1: Prerequisites - [Chapter 1: Prerequisites](https://www.baseten.co/inference-engineering/book/01-prerequisites.md): The framing decisions that come before optimization: app requirements, model selection, and performance measurement. - [1.1 Scale and Specialization](https://www.baseten.co/inference-engineering/book/01-prerequisites/1.1-scale-and-specialization.md): The two paths to adding AI to a product, shared model APIs versus dedicated deployments, and when to switch. - [1.2 About Your App](https://www.baseten.co/inference-engineering/book/01-prerequisites/1.2-about-your-app.md): How modality, latency budget, unit economics, and traffic patterns shape every downstream inference decision. - [1.3 Model Selection](https://www.baseten.co/inference-engineering/book/01-prerequisites/1.3-model-selection.md): Picking a model through evaluation, then using fine-tuning or distillation to hit quality targets with something smaller. - [1.4 Measuring Latency and Throughput](https://www.baseten.co/inference-engineering/book/01-prerequisites/1.4-measuring-latency-and-throughput.md): TTFT, tokens per second, latency percentiles, and end-to-end metrics: defining performance before optimizing it. ## Chapter 2: Models - [Chapter 2: Models](https://www.baseten.co/inference-engineering/book/02-models.md): How LLMs and image generation models actually compute, and where their inference bottlenecks come from. - [2.1 Neural Networks](https://www.baseten.co/inference-engineering/book/02-models/2.1-neural-networks.md): Linear layers, matrix multiplication, and activation functions as the compute primitives behind generative models. - [2.2 LLM Inference Mechanics](https://www.baseten.co/inference-engineering/book/02-models/2.2-llm-inference-mechanics.md): Autoregressive generation, prefill and decode, transformer blocks, attention, and mixture of experts architectures. - [2.3 Image Generation Inference Mechanics](https://www.baseten.co/inference-engineering/book/02-models/2.3-image-generation-inference-mechanics.md): Diffusion pipelines, few-step image models, and how video generation extends the same mechanics. - [2.4 Calculating Inference Bottlenecks](https://www.baseten.co/inference-engineering/book/02-models/2.4-calculating-inference-bottlenecks.md): Using ops:byte ratio and arithmetic intensity to determine whether a workload is compute bound or memory bound. - [2.5 Optimizing Attention](https://www.baseten.co/inference-engineering/book/02-models/2.5-optimizing-attention.md): Why attention scales quadratically with sequence length, and the algorithms that make long context tractable. ## Chapter 3: Hardware - [Chapter 3: Hardware](https://www.baseten.co/inference-engineering/book/03-hardware.md): GPU architecture, NVIDIA generations from Hopper to Rubin, multi-GPU instances, alternative accelerators, and local inference. - [3.1 GPU Architecture](https://www.baseten.co/inference-engineering/book/03-hardware/3.1-gpu-architecture.md): Streaming multiprocessors, tensor cores, and the memory hierarchy from HBM through caches to registers. - [3.2 GPU Architecture Generations](https://www.baseten.co/inference-engineering/book/03-hardware/3.2-gpu-architecture-generations.md): What changed across Hopper, Ada Lovelace, Blackwell, and Rubin, plus the Grace and Vera CPUs paired with them. - [3.3 Instances](https://www.baseten.co/inference-engineering/book/03-hardware/3.3-instances.md): Multi-GPU instances, NVLink and InfiniBand interconnect, and slicing a card with multi-instance GPU. - [3.4 Other Datacenter Accelerator Options](https://www.baseten.co/inference-engineering/book/03-hardware/3.4-other-datacenter-accelerator-options.md): AMD GPUs, TPUs, and custom inference silicon as alternatives to NVIDIA hardware. - [3.5 Local Inference](https://www.baseten.co/inference-engineering/book/03-hardware/3.5-local-inference.md): Running models on desktops and phones, and the constraints that separate edge inference from the datacenter. ## Chapter 4: Software - [Chapter 4: Software](https://www.baseten.co/inference-engineering/book/04-software.md): The inference software stack, from CUDA kernels up through PyTorch, vLLM, SGLang, TensorRT-LLM, and Dynamo. - [4.1 CUDA](https://www.baseten.co/inference-engineering/book/04-software/4.1-cuda.md): Writing and selecting CUDA kernels for inference, and cutting memory accesses through kernel fusion. - [4.2 Deep Learning Frameworks and Libraries](https://www.baseten.co/inference-engineering/book/04-software/4.2-deep-learning-frameworks-and-libraries.md): PyTorch and the libraries that sit between writing raw CUDA and adopting an off-the-shelf inference engine. - [4.3 Inference Engines](https://www.baseten.co/inference-engineering/book/04-software/4.3-inference-engines.md): Comparing vLLM, SGLang, and TensorRT-LLM, and how to choose between them for a given workload. - [4.4 NVIDIA Dynamo](https://www.baseten.co/inference-engineering/book/04-software/4.4-nvidia-dynamo.md): Dynamo as a distributed serving layer for multi-node, disaggregated, and cache-aware inference. - [4.5 Performance Benchmarking and Load Testing](https://www.baseten.co/inference-engineering/book/04-software/4.5-performance-benchmarking-and-load-testing.md): Building benchmarks that reflect real production traffic instead of producing misleading numbers. ## Chapter 5: Techniques - [Chapter 5: Techniques](https://www.baseten.co/inference-engineering/book/05-techniques.md): The core optimization techniques: quantization, speculative decoding, caching, parallelism, and disaggregation. - [5.1 Quantization](https://www.baseten.co/inference-engineering/book/05-techniques/5.1-quantization.md): Lowering precision to FP8, FP4, and beyond to cut memory traffic and unlock compute without losing quality. - [5.2 Speculative Decoding](https://www.baseten.co/inference-engineering/book/05-techniques/5.2-speculative-decoding.md): Draft tokens, validation, and acceptance rate: generating more than one token per forward pass. - [5.3 Caching](https://www.baseten.co/inference-engineering/book/05-techniques/5.3-caching.md): KV cache mechanics, prefix caching across requests, and cache-aware routing between replicas. - [5.4 Model Parallelism](https://www.baseten.co/inference-engineering/book/05-techniques/5.4-model-parallelism.md): Tensor, pipeline, expert, and data parallelism for models too large to serve on a single GPU. - [5.5 Disaggregation](https://www.baseten.co/inference-engineering/book/05-techniques/5.5-disaggregation.md): Separating prefill and decode onto independently scaling workers, and when that pays off. ## Chapter 6: Modalities - [Chapter 6: Modalities](https://www.baseten.co/inference-engineering/book/06-modalities.md): Inference characteristics and optimizations for vision, embedding, speech, image, and video models. - [6.1 Vision Language Models](https://www.baseten.co/inference-engineering/book/06-modalities/6.1-vision-language-models.md): Serving models that accept images and video alongside text, and the encoder costs that come with them. - [6.2 Embedding Models](https://www.baseten.co/inference-engineering/book/06-modalities/6.2-embedding-models.md): High-throughput embedding inference for search, RAG, and semantic similarity workloads. - [6.3 ASR Models](https://www.baseten.co/inference-engineering/book/06-modalities/6.3-asr-models.md): Automatic speech recognition for transcription and dictation, including streaming audio and Whisper-family serving. - [6.4 TTS Models](https://www.baseten.co/inference-engineering/book/06-modalities/6.4-tts-models.md): Speech synthesis under the latency requirements of real-time voice agents. - [6.5 Image Generation Models](https://www.baseten.co/inference-engineering/book/06-modalities/6.5-image-generation-models.md): How serving image generation differs from LLM serving, and the optimizations specific to it. - [6.6 Video Generation Models](https://www.baseten.co/inference-engineering/book/06-modalities/6.6-video-generation-models.md): The most computationally demanding modality, and what it takes to serve it in production. ## Chapter 7: Production - [Chapter 7: Production](https://www.baseten.co/inference-engineering/book/07-production.md): Running inference as a production system: containers, autoscaling, multi-cloud capacity, testing, and client code. - [7.1 Containerization](https://www.baseten.co/inference-engineering/book/07-production/7.1-containerization.md): Packaging model servers so deployments are reproducible and start quickly. - [7.2 Autoscaling](https://www.baseten.co/inference-engineering/book/07-production/7.2-autoscaling.md): Scaling replicas to meet traffic and latency SLAs without paying for idle GPUs, including cold start mitigation. - [7.3 Multi-Cloud Capacity Management](https://www.baseten.co/inference-engineering/book/07-production/7.3-multi-cloud-capacity-management.md): Treating capacity across regions and cloud providers as one elastic pool instead of isolated silos. - [7.4 Testing and Deployment](https://www.baseten.co/inference-engineering/book/07-production/7.4-testing-and-deployment.md): End-to-end testing, canary and blue-green rollouts, and safe deploys for model servers. - [7.5 Client Code](https://www.baseten.co/inference-engineering/book/07-production/7.5-client-code.md): The application-side patterns for calling inference endpoints reliably. - [7.6 Production Inference with Baseten](https://www.baseten.co/inference-engineering/book/07-production/7.6-production-inference-with-baseten.md): How the concepts in this book map onto Baseten's inference platform. ## Appendices - [Appendix A: Inference Glossary](https://www.baseten.co/inference-engineering/book/appendix-a-inference-glossary.md): Alphabetized definitions of the essential terms used throughout the book. - [Appendix B: Recommended Reading](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading.md): Curated papers, documentation, books, and blogs for going deeper, organized by topic. - [Architecture](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading/architecture.md): Foundational papers on transformers, attention, diffusion, and model architecture. - [Developer Tools](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading/developer-tools.md): Documentation and tooling for building, serving, and optimizing models. - [Frontier Open Models](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading/frontier-open-models.md): Papers and technical reports for the leading open models. - [GPU Infrastructure](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading/gpu-infrastructure.md): References on GPU hardware, interconnect, and datacenter infrastructure. - [Inference Optimization Research](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading/inference-optimization-research.md): Research behind quantization, speculation, caching, attention, and parallelism. - [Intelligence Evaluation](https://www.baseten.co/inference-engineering/book/appendix-b-recommended-reading/intelligence-evaluation.md): Benchmarks and evaluation methodology for measuring model quality. ## Optional - [Complete book text](https://www.baseten.co/inference-engineering/llms-full.txt): Every chapter concatenated into one file, roughly 90,000 tokens. Prefer the individual chapters above. - [Acknowledgements](https://www.baseten.co/inference-engineering/book/acknowledgements.md): Credits and thanks to the reviewers, editors, and illustrators behind the book. - [Book landing page](https://www.baseten.co/inference-engineering/): Order a print copy or download the ebook.