This book is an introduction to the field of inference engineering. There is endless depth to explore in every one of the technologies and techniques behind performant inference at scale.
If you’re in the market for another book to continue learning, I have three recommendations:
- AI Engineering: Building Applications with Foundation Models by Chip Huyen (O’Reilly Media, 2025): This incredibly popular book introduces the full breadth of AI engineering topics.
- Build a Large Language Model (From Scratch) by Sebastian Raschka (Manning, 2024): This hands-on book provides a detailed look at LLM architecture.
- AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch by Chris Fregly (O’Reilly Media, 2025): This brand-new book focuses on building for performance.
The AI industry moves fast, and new models, research, and implementations are constantly being released. My colleagues and I publish our latest work on the Baseten blog, which you can access at https://www.baseten.com/blog.
This appendix provides a list of papers, documentation, books, and blogs to further support your next steps as an inference engineer. Resources are organized by topic and alphabetized by title within each section.
Architecture
- “Attention is All You Need,” by Ashish Vaswani et al. (Neural Information Processing Systems, 2017), https://arxiv.org/abs/1706.03762
- “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” by Jacob Devlin et al. (North American Chapter of the Association for Computational Linguistics, 2019), https://arxiv.org/abs/1810.04805
- “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” by Junnan Li et al. (International Conference on Machine Learning, 2023), https://arxiv.org/abs/2301.12597
- Deep Learning, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (The MIT Press, 2016), https://www.deeplearningbook.org/
- Deep Learning with Python (2nd Edition), by François Chollet (Manning, 2021), https://www.manning.com/books/deep-learning-with-python-second-edition
- “Denoising Diffusion Probabilistic Models,” by Jonathan Ho et al. (ArXiv abs/2006.11239, 2020), https://arxiv.org/abs/2006.11239
- “DiT: Scalable Diffusion Models with Transformers,” by William Peebles and Saining Xie (International Conference on Computer Vision (ICCV), 2022), https://arxiv.org/abs/2212.09748
- “FlashAttention: Fast and Memory-Efficient Exact Attention with IO Awareness,” by Tri Dao et al. (ArXiv abs/2205.14135, 2022), https://arxiv.org/abs/2205.14135
- “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” by Tri Dao (ArXiv abs/2307.08691, 2023), https://arxiv.org/abs/2307.08691
- “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision,” by Jay Shah et al. (ArXiv abs/2407.08608, 2024), https://arxiv.org/abs/2407.08608
- Flash-Attention-4, by Tri Dao (Dao AI Research Lab, 2025), https://github.com/Dao-AILab/flash-attention
- “Imagen Video: High Definition Video Generation with Diffusion Models,” by Jonathan Ho et al. (ArXiv abs/2210.02303, 2022), https://arxiv.org/abs/2210.02303
- “Language Models Are Few-Shot Learners,” by Tom Brown et al. (ArXiv abs/2005.14165), https://arxiv.org/abs/2005.14165
- “Learning Transferable Visual Models from Natural Language Supervision,” by Alec Radford et al. (International Conference on Machine Learning, 2021), https://arxiv.org/abs/2103.00020
- “Longformer: The Long-Document Transformer,” by Iz Beltagy et al. (ArXiv abs/2004.05150, 2020), https://arxiv.org/abs/2004.05150
- “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” by Albert Gu and Tri Dao (ArXiv abs/2312.00752, 2023), https://arxiv.org/abs/2312.00752
- “Matryoshka Representation Learning,” by Aditya Kusupati et al. (Neural Information Processing Systems, 2022), https://arxiv.org/abs/2205.13147
- “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” by Noam Shazeer et al. (ArXiv abs/1701.06538, 2017), https://arxiv.org/abs/1701.06538
- “Reformer: The Efficient Transformer,” by Nikita Kitaev et al. (ArXiv abs/2001.04451, 2020), https://arxiv.org/abs/2001.04451
- “Robust Speech Recognition via Large-Scale Weak Supervision,” by Alec Radford et al. (International Conference on Machine Learning, 2022), https://arxiv.org/abs/2212.04356
- “RoFormer: Enhanced Transformer with Rotary Position Embedding,” by Jianlin Su et al. (ArXiv abs/2104.09864, 2021), https://arxiv.org/abs/2104.09864
- “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis,” by Dustin Podell et al. (ArXiv abs/2307.01952, 2023), https://arxiv.org/abs/2307.01952
- “Segment Anything,” by Alexander Kirillov et al. (2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023), https://arxiv.org/abs/2304.02643
- “Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks,” by Nils Reimers and Iryna Gurevych (ArXiv abs/1908.10084, 2019), https://arxiv.org/abs/1908.10084
- “The Llama 3 Herd of Models,” by Aaron Grattafiori et al. (ArXiv 2407.21783, 2024), https://arxiv.org/abs/2407.21783
- “Video Diffusion Models,” by Jonathan Ho et al. (ArXiv abs/2204.03458, 2022), https://arxiv.org/abs/2204.03458
- “Visual Instruction Tuning,” by Haotian Liu et al. (ArXiv abs/2304.08485, 2023), https://arxiv.org/abs/2304.08485
Developer Tools
- BitsAndBytes by bitsandbytes-foundation, https://github.com/TimDettmers/bitsandbytes
- ComfyUI by comfyanonymous, https://github.com/comfyanonymous/ComfyUI
- “CUDA by Example: An Introduction to General Purpose GPU Programming,” by Jason Sanders and Edward Kandrot (NVIDIA developer, 2025), https://developer.nvidia.com/cuda-example
- “CUDA C++ Programming Guide Release 13.0,” by NVIDIA (2025), https://docs.nvidia.com/cuda/cuda-c-programming-guide/
- “CUDA cuBLAS Release 13.0,” by NVIDIA (2025),https://docs.nvidia.com/cuda/cublas/
- CUTLASS by NVIDIA, https://github.com/NVIDIA/cutlass
- DeepGEMM by DeepSeek-ai, https://github.com/deepseek-ai/DeepGEMM
- Hugging Face Diffusers by Hugging Face, https://huggingface.co/docs/diffusers/index
- LMCache by LMCache Project, https://github.com/LMCache/LMCache
- “NVIDIA Dynamo Documentation,” by NVIDIA (2025), https://docs.nvidia.com/dynamo/latest/index.html
- NVIDIA Nsight Systems by NVIDIA, https://developer.nvidia.com/nsight-systems
- NVIDIA Triton Inference Server by NVIDIA, https://github.com/triton-inference-server/server
- ONNX Runtime by Microsoft, https://onnxruntime.ai/
- “PyTorch Performance Tuning Guide,” by Szymon Migacz (PyTorch Foundation, 2020), https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html
- “PyTorch Profiler,” by Shivam Raikundalia (PyTorch Foundation, 2021), https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
- SGLang Project by LMSYS Org, https://github.com/sgl-project/sglang
- “TensorRT Documentation,” by NVIDIA (2025), https://docs.nvidia.com/deeplearning/tensorrt/
- TensorRT-LLM by NVIDIA, https://github.com/NVIDIA/TensorRT-LLM
- Transformers by Hugging Face, https://huggingface.co/docs/transformers/index
- vLLM Project by The Linux Foundation, https://github.com/vllm-project/vllm
Frontier Open Models
- DeepSeek, by DeepSeek AI, https://huggingface.co/deepseek-ai
- FLUX, by Black Forest Labs**,** https://huggingface.co/black-forest-labs
- Gemma, by Google, https://huggingface.co/google
- GLM, by Z.ai, https://huggingface.co/zai-org
- GPT OSS, by OpenAI, https://huggingface.co/openai
- Kimi, by Moonshot AI, https://huggingface.co/moonshotai
- Llama, by Meta Llama, https://huggingface.co/meta-llama
- MiniMax, by MiniMax AI, https://huggingface.co/MiniMaxAI
- Mistral, by Mistral AI, https://huggingface.co/mistralai
- Nemotron, by NVIDIA, https://huggingface.co/nvidia
- Orpheus, by Canopy Labs, https://huggingface.co/canopylabs
- Qwen, by Alibaba Qwen, https://huggingface.co/Qwen
- Wan, by Wan-AI, https://huggingface.co/Wan-AI
- Whisper, by OpenAI, https://huggingface.co/openai
GPU Infrastructure
- Designing Data-Intensive Applications, by Martin Kleppmann (O’Reilly Media, 2017) https://dataintensive.net/
- Grace Hopper / Grace Blackwell Systems by NVIDIA, https://www.nvidia.com/en-us/data-center/grace-cpu/
- GPU Glossary, by Frye et al. (Modal, 2025), https://modal.com/gpu-glossary
- InfiniBand, by NVIDIA, https://www.nvidia.com/en-us/networking/products/infiniband/
- Kubernetes Documentation, by The Kubernetes Authors (The Linux Foundation, 2025), https://kubernetes.io/docs/home/
- NVIDIA Blackwell Architecture Technical Brief: Built for the Age of AI Reasoning, (NVIDIA, 2025), https://resources.nvidia.com/en-us-blackwell-architecture?ncid=no-ncid
- NVIDIA H100 Tensor Core GPU Architecture: Exceptional Performance, Scalability and Security for the Data Center, (NVIDIA, 2023), https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c
- "NVIDIA Tesla: A Unified Graphics and Computing Architecture," by E. Lindholm et al. (IEEE Micro, March–April 2008), https://ieeexplore.ieee.org/document/4523358
- NVLink / NVSwitch, by NVIDIA, https://www.nvidia.com/en-us/data-center/nvlink/
- Programming Massively Parallel Processors: A Hands-on Approach, by Wen-mei Hwu, David Kirk, Izzat El Hajj (Morgan Kaufmann, 2022), https://www.goodreads.com/work/editions/10244675-programming-massively-parallel-processors-a-hands-on-approach
- SemiAnalysis, by Dylan Patel (SemiAnalysis, 2025), https://semianalysis.com/
- Site Reliability Engineering: How Google Runs Production Services, edited by Betsy Beyer et al. (O’Reilly Media, 2017), https://sre.google/books/
Inference Optimization Research
- “Adversarial Diffusion Distillation,” by Axel Sauer et al. (European Conference on Computer Vision, 2023), https://arxiv.org/abs/2311.17042
- “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” by Ofir Press, Noah Smith, and Mike Lewis (ArXiv abs/2108.12409, 2021), https://arxiv.org/abs/2108.12409
- “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,” by Song Han (MIT, 2024), https://github.com/mit-han-lab/llm-awq
- Cache-DIT by Vipshop, https://github.com/vipshop/cache-dit
- “CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion,” by Jiayi Yao et al. (Proceedings of the Twentieth European Conference on Computer Systems, 2024), https://arxiv.org/abs/2405.16444
- “Adding Conditional Control to Text-to-Image Diffusion Models,” by Lymin Zhang et al. (International Conference on Computer Vision, 2023), https://arxiv.org/abs/2302.05543
- “Beyond the Buzz: A Pragmatic Take on Inference Disaggregation,” by Tiyasa Mitra et al. (ArXiv abs/2506.05508, 2025), https://arxiv.org/abs/2506.05508
- “Break the Sequential Dependency of LLM Inference Using Lookahead Decoding,” by Yichao Fu et al. (ArXiv abs/2402.02057, 2024), https://arxiv.org/abs/2402.02057
- “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” by Yuhui Li et al. (ArXiv abs/2401.15077, 2024), https://arxiv.org/abs/2401.15077
- “EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees,” by Yuhui Li et al. (Conference on Empirical Methods in Natural Language Processing, 2024), https://arxiv.org/abs/2406.16858
- “EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test,” by Yuhui Li et al. (ArXiv abs/2503.01840, 2025), https://arxiv.org/abs/2503.01840
- “FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving,” by Ye et al. (ArXiv abs/2501.01005, 2025), https://arxiv.org/abs/2501.01005
- “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” by Elias Frantar (ArXiv abs/2210.17323, 2022), https://arxiv.org/abs/2210.17323
- “High-Resolution Image Synthesis with Latent Diffusion Models,” by Robin Rombach et al. (Conference on Computer Vision and Pattern Recognition (CVPR), 2021), https://arxiv.org/abs/2112.10752
- “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” by Simian Luo (ArXiv abs/2310.04378, 2023), https://arxiv.org/abs/2310.04378
- “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” by Tim Dettmers et al. (ArXiv abs/2208.07339, 2022), https://arxiv.org/abs/2208.07339
- “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” by Tianle Cai et al. (ArXiv abs/2401.10774, 2024), https://arxiv.org/abs/2401.10774
- “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” by Mohammad Shoeybi et al. (ArXiv abs/1909.08053, 2019), https://arxiv.org/abs/1909.08053
- “Efficient Memory Management for Large Language Model Serving with PagedAttention,” by Woosuk Kwon et al. (Proceedings of the 29th Symposium on Operating Systems Principles, 2023), https://arxiv.org/abs/2309.06180
- “Fast Inference from Transformers via Speculative Decoding,” by Yaniv Leviathan et al. (International Conference on Machine Learning, 2022), https://arxiv.org/abs/2211.17192
- “Ring Attention with Blockwise Transformers for Near-Infinite Context,” by Hao Lin et al. (ArXiv abs/2310.01889, 2023), https://arxiv.org/abs/2310.01889
- “SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration,” by Jintao Zhang et al. (ArXiv abs/2410.02367, 2024), https://arxiv.org/abs/2410.02367
- Sequence/Context Parallelism, by Megatron-LM for NVIDIA https://github.com/NVIDIA/Megatron-LM
- SmoothQuant by Song Han (MIT), https://github.com/mit-han-lab/smoothquant
- “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot,” by Elias Frantar and Dan Alistarh, (ArXiv abs/2301.00774, 2023), https://arxiv.org/abs/2301.00774
- “SpecVLM: Fast Speculative Decoding in Vision-Language Models,” Haiduo Huang et al. (ArXiv abs/2509.11815, 2025), https://arxiv.org/abs/2509.11815
- “TeaCache: Efficient KV Cache Compression via Tensor Decomposition,” by Feng Lu et al. (Alibaba TongYi Vision Intelligence Lab, ArXiv abs/2411.19108, 2025), https://github.com/ali-vilab/TeaCache
Intelligence Evaluation
- ARC AGI Prize by Greg Kamradt (2025), https://arcprize.org/
- Evals for AI Engineers: Systematically Measuring and Improving AI Applications, by Shreya Shankar and Hamel Husain (O’Reilly Media, forthcoming 2026) https://www.oreilly.com/library/view/evals-for-ai/9798341660717/
- Grade School Math: Training Verifiers to Solve Math Word Problems, by Karl Cobbe and Vineet Kosaraju (ArXiv abs/2110.14168, 2021), https://github.com/openai/grade-school-math
- “How to Fine-Tune Qwen3 to GPT-4o Level Performance,” by Greg Schoeninger (Fine-Tune Fridays, Oxen AI, 2025), https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o-level-performance/
- “Humanity's Last Exam,” by Long Phan et al. (Center for AI Safety and Scale AI, ArXiv abs/2501.14249, 2025), https://www.lastexam.ai/
- “HumanEval: Evaluating Large Language Models Trained on Code,” by Michelle Pokrass, Qiming Yuan, and Yichen Xu (OpenAI, 2021), https://github.com/openai/human-eval
- “MMLU: Measuring Massive Multitask Language Understanding,” by Dan Hendrycks et al. (Proceedings of the International Conference on Learning Representations (ICLR), 2021), https://github.com/hendrycks/test
- “MTEB: Massive Text Embedding Benchmark,” by Niklas Muenninghoff et al. (Conference of the European Chapter of the Association for Computational Linguistics, 2022), https://arxiv.org/abs/2210.07316
- “SWE-Bench: Can Language Models Resolve Real-World Github Issues?” by Carlos Jimenez et al. (Proceedings of the International Conference on Learning Representations (ICLR), 2024), https://www.swebench.com/