# Chapter 5: Techniques _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) One of the coolest parts about working in inference engineering is that unlike many industries where new academic research takes years or decades to be adopted by industry, techniques from new papers are live in production within months or even weeks. There is a gap to cross between research and production, and some of the most visible inference engineering work in the industry comes from bridging that gap. A core principle of inference engineering is that the more constraints you can introduce in your inference system, the better performance you’ll achieve. This principle continues to apply throughout this chapter, with techniques like disaggregation, which allows you to constrain individual engines to prefill and decode. With these model performance techniques, there’s a new principle to keep in mind: the more traffic you have, the more performance optimizations you can make (while keeping unit economics reasonable). Higher model parallelism across more GPUs, KV-aware routing, and dynamic disaggregation only make sense when you have a large number of GPUs, often multiple nodes, serving the same model with vertical scale and horizontal replication. Real-world traffic defies constraints. But with volume, you can adapt your systems over time to match the changing nature of usage. Tuning the parameters of inference engines, speculation algorithms, and model servers isn’t a one-time task. Instead, either through iterative deployments or dynamic runtime adjustments, you can continuously improve the performance of your inference system. Finding the right combination of techniques and configurations takes patient experimentation. I remember an internal hackathon during which one of Baseten’s inference engineers was working on an autocomplete model for code and ended up trying 77 different configurations via a handwritten script before finding a non-obvious solution that doubled TPS for a customer’s model. To make inference optimization even more complex, sometimes techniques are symbiotic and sometimes they are incompatible. For example, quantizing the KV cache alleviates a bottleneck in disaggregation, but increasing batch sizing reduces the compute available for speculation. An inference engineer’s goal is always to create a balanced set of optimizations that delivers more than the sum of its parts. This chapter introduces five key categories of applied research for inference acceleration: quantization, speculation, caching, parallelism, and disaggregation. In each section, pay special attention to the recommended circumstances for using each technique and the potential bottlenecks or tradeoffs each introduces. ## 5.1 Quantization Quantization improves latency (both TTFT and TPS), increases system throughput, and opens up headroom for other optimizations like disaggregation, speculation, and prefix caching to be even more effective. But when it goes wrong, quantization can materially reduce the model’s output quality. Models are trained with weights, activations, and other components represented in a certain native number format. Usually, this is BF16 or FP16, though 8-bit and 4-bit native precisions are becoming more popular in training. Post-training quantization works by changing those model weights and other values from their native number format to a lower-precision format. Cutting precision in half improves performance in both phases of inference: - **Prefill:** Compute-bound prefill now runs on lower-precision Tensor Cores with twice the FLOPS. - **Decode:** Memory-bound decode now loads half as much data per value, effectively doubling memory bandwidth. Working with quantized data does introduce overhead, so it’s not linearly twice as fast to go from 16 to 8 bits. In practice, quantization down a single level of precision generally offers 30 to 50 percent better performance for LLMs. The catch with quantization is that it runs the risk of reducing the model’s output quality. Quantization has the potential to introduce precision errors throughout the calculations that power inference. Precision errors compound over time. Consider what happens when you square and cube different precisions of Pi: | Pi precision | Pi squared | Pi cubed | | :----------- | :--------- | :-------- | | 3.14159 | 9.869588 | 31.006198 | | 3.14 | 9.8596 | 30.959144 | | 3 | 9 | 27 | Most of the work in quantization is around both preventing precision errors and minimizing their impact on the final model output. ### 5.1.1 Number Formats Quantization introduces a new collection of essential terms and abbreviations. The most important ones to know are the common number formats: | Name | Abbr | First architecture | | :---------------------- | :---- | :----------------------------- | | 64-bit Floating Point | FP64 | Fermi (2010) | | 32-bit Floating Point | FP32 | Kepler (2012) | | 16-bit Floating Point | FP16 | Pascal (2016) | | Brain Floating Point 16 | BF16 | Ampere (2020) | | 8-bit Floating Point | FP8 | Hopper (2022) | | Mixed-Precision FP8 | MXFP8 | Blackwell (2024) | | 8-bit Integer | INT8 | Pascal (2016) | | 6-bit Floating Point | FP6 | Blackwell (2024, experimental) | | 4-bit Floating Point | FP4 | Blackwell (2024) | | Mixed-Precision FP4 | MXFP4 | Blackwell (2024) | | NVIDIA FP4 | NVFP4 | Blackwell (2024, proprietary) | | 4-bit Integer | INT4 | Turing (2018) | The largest number format, FP64 or “double precision,” is only used for high-precision scientific computing, not AI training or inference. FP32 is sometimes used for training, but almost never for inference. FP6 is more experimental at the time of publication, though AMD GPUs are rapidly adopting the format. That leaves 16, 8, and 4-bit precisions as the primary formats for inference. Number formats have a: - **Precision:** The number of bits used to express a single value in the format. For example, FP16 uses 16 bits. - **Type:** Whether these bits are interpreted to represent an integer (no decimal) or a floating-point number (a decimal). - **Scale factor:** A multiplier used to map values from a low-precision format back to the higher-precision format. Together, these attributes determine the two factors behind how well a number format represents values used in inference: - **Dynamic range:** The difference between the lowest and highest value that can be represented in the format. - **Granularity:** The number of parameters or other values that are quantized along a single scale factor. Dynamic range is essential to low-precision inference without quality loss. 16 bits can represent 65,536 distinct values, while 8 bits can only represent 256 different values. The dynamic range is the distribution of these values – the difference between the smallest and largest available value. Dynamic range explains why floating-point formats are better than integer formats for inference. Floating-point formats have three properties: - **Sign:** A single bit that represents whether the number is positive or negative. - **Exponent:** A set of bits that, taken together, represent an exponent factor. - **Mantissa:** A set of bits that, taken together, represent the base value multiplied by two to the exponent. An FP8 number in a E4M3 data format means it has a 4-bit exponent and a 3-bit mantissa, with the remaining bit for the sign. Integer formats only have sign and value bits. ![Figure 5.1: Floating-point number formats have both exponent and mantissa bits along with the sign bit.](https://www.datocms-assets.com/104802/1788123373-inference-engineering-figure-5-1.png) _Figure 5.1: Floating-point number formats have both exponent and mantissa bits along with the sign bit._ The exponent in floating-point numbers gives it a higher dynamic range, meaning it can better express very large and very small numbers. This is important because outlier values are significant in inference, and floating-point number formats better represent outliers after quantization. Within floating-point formats, there are multiple options at each precision, like FP4, MXFP4, and NVFP4. These formats differ in granularity, or the number of values that are quantized by a single scale factor. Quantization can be applied at the: - **Tensor level:** Calculate a single scale factor for the entire QKV tensor. - **Channel level:** Calculate a different scale factor for each feature vector within the tensor. - **Block level:** Within each feature vector, divide the vector into blocks of N values and calculate a scale factor for each block. More granular quantization has a lower chance of smoothing over outliers, preserving quality. However, more granularity introduces more overhead for storing and applying scale factors. MXFP8 and MXFP4, the new number formats supported by Blackwell, are “microscaling” formats that compute a blockwise scale factor on every 32 parameters, reducing the impact of these number formats’ lower dynamic range. NVFP4, a 4-bit format by NVIDIA, offers even higher granularity than the MX formats with a block size of 16 and a secondary 32-bit global scale factor to further combat the quality loss that 4-bit formats introduce. The tradeoff to a microscaling format is that the small-block scale factor must also be stored in memory, slightly reducing the performance gains from quantization. Additionally, both the tensor and block scale factors need to be applied, introducing a bit of compute overhead. Blackwell GPUs offset this overhead via scale factor application in Tensor Cores. While the focus of this book is on inference engineering in the datacenter, quantization is an essential topic for local and edge inference, especially for large models. GGUF, a binary format for storing models, is the most popular choice for distributing highly quantized models on Hugging Face, with individual researchers and companies squeezing huge models like DeepSeek onto consumer hardware like Apple computers. These quantization strategies combat quality loss through dynamic quantization, where certain layers or other components of the model are left in their original precision, while others are quantized to integers with as little as one bit of precision. Dynamic formats represent their average precision, which is why you might see something like fine-tuning lab Unsloth’s popular 1.58-bit quantization. While these dynamic quantizations are impressive feats of engineering and are great for local inference, inference engineers working on production systems should stick with floating point number formats – integer formats are not suitable for quality-sensitive workloads due to their lack of dynamic range. Instead, 8-bit floating-point formats (FP8, MXFP8) are generally the sweet spot for improving performance without sacrificing quality. FP4 is promising, especially the NVFP4 format which introduces a higher level of granularity for improved accuracy, but FP8 and MXFP8 provide the most flexibility, especially when quantizing the KV cache. ### 5.1.2 Quantization Approaches The more parameters a model has, the less sensitive it is to quantization as each individual parameter is less important. However, even for very large models, it is essential to quantize carefully. Quantization can happen during or after training: - **Quantization-aware training:** Training weights and computing scale factors together to ensure that the final converged weights are accurate at a given precision. - **Post-training quantization:** Converting finished model weights to a new precision by computing scale factors and preserving accuracy via calibration. While some labs release models created with quantization-aware training, like GPT-OSS in MXFP4 and Kimi K2 Thinking in INT4, inference engineers working with open models only have the ability to perform post-training quantization as they are working with finished weights. A leading tool for post-training quantization is NVIDIA TensorRT Model Optimizer (ModelOpt), an open-source library that also supports pruning, distillation, and sparsity. ModelOpt outputs are compatible with all inference engines (vLLM, SGLang, TensorRT-LLM). After picking a precision, there are two decisions to make before doing post-training quantization: 1. What parts of the model (weights, activations, KV cache, attention) should be quantized? 2. What number format offers the appropriate dynamic range and granularity? These decisions turn quantization from a binary choice into a spectrum of tradeoffs around performance and quality. Components of a model have varying sensitivity to quantization. Reducing the precision of more sensitive components runs a higher risk of quality degradation. From least to most sensitive: 1. **Weights:** Specifically the linear layers are least sensitive to quantization. 2. **Activations:** The intermediate output of activation functions are only somewhat sensitive to quantization. Note that the activation functions themselves are rarely quantized as they are such a tiny fraction of the model’s weights. 3. **KV cache:** The cached values from the attention calculation are moderately sensitive to quantization. 4. **Attention:** The attention layers of a model are highly sensitive to quantization, especially equations like softmax. Within each of these components, you can get more selective about quantization. Even in linear layers and activations, which are generally the least sensitive to quantization thanks to their size, early and late layers like the input and output layer of the neural network may be left in their original precision as these layers are more sensitive. While quantizing weights and activations helps performance directly, KV cache quantization gives an additional boost to techniques like prefix caching and disaggregation. The KV cache is a valuable resource. Quantizing it allows inference engines to store more of it in memory and read it more quickly. However, the KV cache for each token is used by each subsequent token. This means precision errors introduced by quantization can compound from token to token. Compounding errors is exactly the reason why attention layers are the riskiest to quantize. Not only is attention very sensitive to dynamic range, but each attention calculation relies on the results of each previous attention calculation. Over a sequence of thousands of tokens, these errors accumulate quickly. All but the most aggressive quantization schemes run functions like softmax in their original precision. ![Figure 5.2: Quantization risk is low for weights and activations, moderate for KV cache, and high for attention.](https://www.datocms-assets.com/104802/1788123381-inference-engineering-figure-5-2.png) _Figure 5.2: Quantization risk is low for weights and activations, moderate for KV cache, and high for attention._ A moderate approach to low-precision inference uses a format like FP8 with high dynamic range – if possible, a microscaling format like MXFP8 – to carefully quantize select linear layers, activations, and often KV cache values. Even with these high dynamic range formats, components of the attention layer are rarely quantized. ### 5.1.3 Measuring Quality Impact The standard for production-ready quantization is zero perceptible quality loss. After quantizing a model, it’s essential to thoroughly test its output quality versus the original precision. There are three methods for checking model quality after quantization: 1. **Perplexity:** Calculate the perplexity score for the quantized model and compare with the original. 2. **Intelligence benchmarks:** Run a standard intelligence benchmark like MMLU or SWE-bench and compare to original scores. 3. **Custom evals:** Run a product-specific evaluation suite on the quantized model and compare to the original weights. In every case, you’re looking for a difference in scores that’s indistinguishable from noise. LLMs are non-deterministic, so scores vary slightly from run to run. The simplest check on quality is perplexity. Rather than asking the model to generate output, perplexity gives the model expected output sequences and calculates the likelihood of the model predicting those tokens. A higher perplexity means a model is more “surprised” by the sequences – not what you want from a model that’s supposed to predict tokens. After quantization, you’re looking for a minimal increase in perplexity. A more comprehensive quality check relies on a public intelligence benchmark or, better yet, a domain-specific eval that matches your expected real-world usage. On evals, you’re looking for a minimal reduction in the quality score. The best way to get the full picture of the impact of quantization is to run all three types of checks and make apples-to-apples comparisons to the original model weights. Remember that quantization is a scale, not a binary decision. You can still get some performance improvements with lower risk of quality loss by quantizing to FP8 instead of FP4, or by quantizing fewer components of the model, like weights-only quantization. If you’re working in a highly sensitive domain and can’t risk model quality, no worries: every other technique in this chapter is lossless in terms of quality. ## 5.2 Speculative Decoding The decode phase of LLM inference is an autoregressive process in which tokens are generated one at a time. The bottleneck on decode is memory bandwidth, with compute sitting idle at low to moderate batch sizes as weights are read from memory. Speculative decoding takes advantage of that spare compute to try to generate multiple tokens per forward pass through the target model. If an inference engine could generate two, three, or even more tokens for each round-trip of weights through memory, it would generate far more tokens per second. Speculative decoding only improves TPS/ITL, not TTFT. There are multiple algorithms for speculative decoding. They share a common mechanism: 1. The speculator generates one or more **draft tokens**. 2. The **target model**, or the underlying model that you’re trying to accelerate, performs **validation** on these tokens to check if they match what the model would have generated. 3. The target model accepts any valid draft tokens and generates an additional token itself, completing the forward pass. This generates N+1 tokens per forward pass, or iteration through the decode loop, where N is the number of accepted draft tokens. Generating draft tokens is not free, it takes both compute and memory. However, it is much faster for a target model to validate a draft token than to generate an original token. If you imagine a sudoku puzzle, solving it is hard, but checking if the solution is correct is very easy. For the target model, generating a token is like solving a sudoku, while validating a draft token is like checking a finished sudoku. The performance uplift from any speculative decoding strategy depends on three factors: 1. **Draft token cost:** The time it takes to generate a draft token. 2. **Draft sequence length:** The number of draft tokens that are generated per forward pass. 3. **Token acceptance rate:** The percentage of draft tokens that are accepted by the target model. Token acceptance rate is high early in the draft sequence, but draft tokens get less reliable deeper in the sequence. ![Figure 5.3: Speculative decoding from draft token generation and validation to prefix acceptance with subsequent token generation.](https://www.datocms-assets.com/104802/1788123073-inference-engineering-figure-0-2.png) _Figure 5.3: Speculative decoding from draft token generation and validation to prefix acceptance with subsequent token generation._ Aim for short, high-percentage sequences because generating and validating tokens, while inexpensive relative to generating tokens in the original model, still comes with meaningful overhead. Additionally, once a single draft token is rejected as wrong, all subsequent tokens in the sequence are also rejected. Working with speculation is interesting because so many factors affect token acceptance rate. The big one is temperature – higher temperatures yield token distributions that are harder to predict, reducing the effectiveness of speculative decoding. But even factors as simple as subject matter can make a difference on acceptance rate if the draft model or additional head used for speculation is better versed in, say, math than history. Another limitation on speculative decoding is that it’s most useful at low batch sizes where there are spare compute cycles. At higher batch sizes, speculative decoding must be dynamically disabled as compute is too saturated to afford verification. Each speculation algorithm navigates these tradeoffs differently, and careful implementation of the right algorithm for the situation can lead to major improvements in TPS. ### 5.2.1 Draft-Target Speculative Decoding The original method of speculative decoding uses two models: - **Draft model:** An additional model that generates speculative draft tokens. - **Target model:** The original model, which now verifies draft tokens in addition to performing ordinary decode. The most important decision when configuring draft-target speculative decoding is which draft model to use. A good draft model has a high token acceptance rate while requiring minimal resources to run. Draft models are most often smaller members of the same family as the target model, as they share tokenizers and behaviors. As a rule of thumb, the draft model should be at least ten times smaller by parameter count than the target model. Fine-tuning or distillation can improve the token acceptance rate of these small models by teaching them to behave more like the target model. Draft-target speculative decoding is a good choice when you want something that is quick to set up out of the box without doing any training or fine-tuning. However, other speculation algorithms generally offer better performance. Draft-target introduces the most overhead of any speculative decoding method. While the draft model is small, the inference engine must store the draft model’s weights, activations, and KV cache in memory, and dedicate compute cycles to draft model prefill. Additionally, the draft and target model must run in coordination so that they don’t compete for resources, though inference engines like TensorRT-LLM handle that model orchestration. ### 5.2.2 Medusa Medusa was one of the first alternatives to draft-target speculation. Medusa addresses the complexity and overhead of operating a draft model to perform speculation by instead fine-tuning the target model to generate additional tokens per forward pass. Fine-tuning a model for Medusa means grafting additional decoder heads onto the model. The ordinary architecture of an LLM contains a single decoder head, but Medusa adds an additional two to four heads that generate sequential draft tokens. Like with draft-target speculation, draft tokens are validated on the next forward pass. ![Figure 5.4: Medusa heads each generate a draft token on top of the token generated by the target model.](https://www.datocms-assets.com/104802/1788123389-inference-engineering-figure-5-4.png) _Figure 5.4: Medusa heads each generate a draft token on top of the token generated by the target model._ Medusa is still limited on draft token count and draft token acceptance rate, and it is not widely used in production today. However, Medusa inspired more popular techniques like EAGLE. ### 5.2.3 EAGLE The main problem with using an off-the-shelf pretrained model as a draft model is that a model like Qwen 0.5B is designed to be a good standalone LLM on cheap hardware, not to speculate draft tokens on a B200. These draft models are inefficient to run and offer a relatively low acceptance rate. EAGLE offers an alternative: a purpose-built draft model trained from scratch to generate sequences of up to eight draft tokens (two times more than Medusa) with a very high acceptance rate. During inference, LLMs accumulate a lot of context about the predicted tokens in the form of hidden states between layers. Traditional draft models don’t have access to this information. EAGLE is a draft model trained to accept hidden states as input and generate speculative tokens as output. Specifically, it is trained on a set of three hidden states: one from an early layer, one from a middle layer, and one from a late layer. EAGLE is often less than one billion parameters and scales well when given additional training data. ![Figure 5.5: EAGLE speculative models take hidden states as input and produce draft tokens as output.](https://www.datocms-assets.com/104802/1788123397-inference-engineering-figure-5-5.png) _Figure 5.5: EAGLE speculative models take hidden states as input and produce draft tokens as output._ In practice, when using EAGLE with an inference engine like TensorRT-LLM, implementations tend to be straightforward, with post-training EAGLE creation and single-sequence speculation. EAGLE can be attached to the same module (PyTorch class) as the target model, so each forward pass runs inference on both the target model and the EAGLE speculator. This unified pipeline solves the other problem with draft-target decoding, where multiple round trips to the CPU were needed to orchestrate the draft and target models. EAGLE is the go-to speculation algorithm for general use among inference engineers with the knowledge and means to train EAGLE heads and is well-supported by inference engines. Like other speculation techniques, adopting EAGLE for improved latency requires reduced batch sizes, lowering throughput and increasing cost. ### 5.2.4 N-gram Speculation and Lookahead Decoding N-gram speculation uses a different mechanism than other types of speculation. There is no draft model. Instead, in parallel with generating the KV cache, the inference engine constructs an n-gram dictionary. The n-gram dictionary maps a single starting token to an observed sequence of N tokens (the n-gram). ![Figure 5.6: An n-gram dictionary matches prefixes to likely suffixes and is especially useful for constrained languages like code.](https://www.datocms-assets.com/104802/1788123402-inference-engineering-figure-5-6.png) _Figure 5.6: An n-gram dictionary matches prefixes to likely suffixes and is especially useful for constrained languages like code._ This n-gram dictionary contains common sequences from the input text and is first constructed during prefill. During decode, the generated token is fed into the dictionary, and any available suffix is selected as draft tokens. In the next forward pass, the target model verifies these draft tokens as normal. The advantage of n-gram speculation versus EAGLE is that the sequences can be much longer. While EAGLE can generate eight or so draft tokens with a decent acceptance rate, n-gram sequences can exceed ten tokens. However, the acceptance rate for n-grams is only high when the contents of the model output are similar to the model input. N-gram speculation is mostly used for code completion and code revision, where syntax is predictable and output closely matches input. However, within this specific domain, it easily outperforms EAGLE. A similar method to n-gram speculation is Lookahead Decoding, which generates n-grams during inference to fill the dictionary. Lookahead Decoding is more general than n-gram speculation as it doesn’t rely as much on highly repetitive context, but it requires extra compute to generate the n-grams. Every speculative decoding algorithm aims to reduce the total number of forward passes needed to generate a complete output sequence, improving overall latency and specifically tokens per second per user during decode. N-gram speculation excels at code completion and similar tasks, while Lookahead Decoding generalizes in systems with excess compute. ## 5.3 Caching During prefill, the inference engine builds a KV cache (a store of keys and values for each token) on the input sequence. It then updates the KV cache for each token during decode. As inference is autoregressive, the value for each new token depends on the value of every previous token in the sequence. Every inference engine uses KV caching by default on a request-by-request basis. Without KV caching, LLM inference would be unbearably slow as each previous value in the entire sequence would need to be re-calculated for each subsequent token. However, engineers can get even more utility from the KV cache by re-using it between requests rather than solely within each inference sequence. ### 5.3.1 Prefix Caching and KV Cache Re-Use Consider the following two prompts, each with four tokens on most tokenizers, in Figure 5.7. ![Figure 5.7: A pair of four-token sequences with two-token matching prefixes.](https://www.datocms-assets.com/104802/1788123411-inference-engineering-figure-5-7.png) _Figure 5.7: A pair of four-token sequences with two-token matching prefixes._ By default, the inference engine has to run prefill on all four tokens of each prompt. But the first tokens of each prompt – “Weather in” – form a shared prefix between the pair. With prefix caching, you can re-use the KV cache from the first request to improve TTFT on the second request by skipping prefill on the first two tokens and reading in the existing KV cache instead. When you see pay-per-token APIs charge less for “cache hit” input tokens than “cache miss” tokens, this is why – re-using cached tokens takes very little compute power or time. As an inference engineer, you can apply the same principle to reduce latency and improve throughput (thus saving money) on your own deployments. Saving two tokens won’t make a big impact on TTFT, but prefix caching can skip prefill on thousands of tokens in certain domains: - **Complex system prompts:** Agents, customer-facing chatbots, RAG scaffolds, and tool calls often feature long, complex system prompts on every call. - **Code completion:** Code completion, code generation, and other coding functions require passing the same thousands of lines of code as shared context. - **Documents and retrieval:** Document summarization, question answering, and retrieval all add repeated context ahead of user prompts. - **Multi-turn conversations:** Ordinary conversations repeat back every message in a chat template, increasing the savings from prefix caching with every turn. Prefix caching works from the start of the input sequence until the first non-repeated token. The fourth token in the weather example, a question mark, is shared between the two input sequences. However, the prefix ends at the first non-repeated token, so the fourth token isn’t read from cache. Because prefixes end at the first unique token, your context engineering determines your TTFT savings. Consider a different approach to the same prompt: ![Figure 5.8: A pair of four-token sequences with no prefix match, the first tokens are different so it doesn’t matter that the next three are the same.](https://www.datocms-assets.com/104802/1788123419-inference-engineering-figure-5-8.png) _Figure 5.8: A pair of four-token sequences with no prefix match, the first tokens are different so it doesn’t matter that the next three are the same._ Here, there is no savings from prefix caching as the very first token differs between the two sequences, even though every subsequent token is the same. To take advantage of prefix caching, ensure that novel tokens are as late in your context as possible. Prefix caching is the dominant form of KV cache re-use because LLMs are autoregressive. Each token influences every subsequent token, so a single novel token changes the way the model represents the rest of the sequence internally, even if the sequences look the same to a human reader. However, there is active research around other kinds of KV cache re-use to overcome this limitation. Caching arbitrary sequences from the middle of prompts requires correcting both positional embeddings and selectively recomputing KV entries to maintain output quality. Tools like CacheBlend and LMCache support non-prefix sequences, expanding the possibilities for KV cache re-use. ### 5.3.2 Where to Store the KV Cache The KV cache is very valuable. But KV caches take up a lot of memory, and GPUs only have limited VRAM. You can configure how much memory your inference engine allocates to KV cache. For example, in TensorRT-LLM, you would set: ![Figure 5.9: Allocating free GPU memory to the KV cache is an essential configuration decision when running inference engines.](https://www.datocms-assets.com/104802/1788123424-inference-engineering-figure-5-9.png) _Figure 5.9: Allocating free GPU memory to the KV cache is an essential configuration decision when running inference engines._ If you’re working on a B200 GPU with 180 GB of VRAM and used 100 GB for model weights and buffers, this would allocate 80 percent of the remaining VRAM, or 64 GB, to KV cache. Once this allocation fills – and it will fill quickly – you’ll have to start deleting saved KV caches, increasing the chance of a cache miss on future requests. To get more room for KV cache, offload from VRAM to other nearby storage. There are four places where you can store KV cache, in descending order of bandwidth to the GPU: | Level | Memory type | Approximate speed | Approximate size | | :---- | :----------------------- | :--------------------------- | :----------------------------- | | G1 | Device Memory (GPU VRAM) | Terabytes per second | 10s to 100s of gigabytes | | G2 | Host Memory (CPU RAM) | 10s to 100s of GB per second | 100s of gigabytes to terabytes | | G3 | Local SSD | 5-10 GB per second | Terabytes | | G4 | Networked SSD | Gigabytes per second | 10s of terabytes | Certain SKUs, like the GB200, come equipped with CPUs and interconnects offering much faster G2 storage making them great for KV cache offloading. NVIDIA Dynamo provides support for KV cache offloading via KVBM (KV Block Manager). KVBM provides APIs for moving KV cache blocks among different levels of memory. As a general rule, you want to keep the most frequently used blocks in higher-bandwidth memory, while less-often-used blocks can be relegated to slower storage until needed. ### 5.3.3 Cache-Aware Routing In a production environment, there will be multiple replicas of your inference server, with incoming traffic split across the replicas. Usually, traffic is routed based on how busy each replica is. If your inference server makes heavy use of prefix caching, your routing logic needs to be updated to account for that. A user in a long conversation with a chatbot or asking multiple questions about a codebase should have their request routed to the same replica whenever possible so that they get a cache hit for a faster, less expensive request. ![Figure 5.10: Cache-aware routing allocates traffic based on KV cache rather than simply dividing requests evenly across replicas.](https://www.datocms-assets.com/104802/1788123429-inference-engineering-figure-5-10.png) _Figure 5.10: Cache-aware routing allocates traffic based on KV cache rather than simply dividing requests evenly across replicas._ Another option is using the G4 networked storage to build a global KV cache across replicas. Routing still matters here – a replica with a hot G1 cache will serve the request faster than a replica reading from G4 – but a global cache ensures that all replicas can eventually access any pre-computed sequence and that cached sequences are not lost when nodes cycle or are spun down during autoscaling. ### 5.3.4 Long Context Handling “Long context” is a bit of a tautological definition: a sequence becomes “long context” when it generates a KV cache large enough to cause problems during inference. Depending on the model, hardware, engine, and traffic, these problems can start to emerge past common cutoffs like 32K, 64K, or 128K tokens. In your performance benchmarking, be sure to send very large input sequences to test your inference service against long context requests. Foundation model labs have been using scaling techniques like RoPE to unlock longer and more accurate context windows. But supporting these upgraded context windows introduces new challenges in inference. Accounting for the KV cache, the attention equation scales linearly with sequence length. With long sequences, attention can become the main consumer of VRAM – the very resource decode is limited by. While approaches like sliding window attention, compressed attention, and sparse attention offer solutions on a model-by-model basis, there are general approaches to optimizing the standard attention algorithm: - **FlashAttention:** A series of optimized attention kernels to compute attention with reduced numbers of reads from and writes to memory. - **PagedAttention:** A memory management technique that stores KV cache in fixed-size pages, reducing fragmentation and duplication. - **Chunked Prefill:** A strategy of splitting large input sequences into chunks, which can be run alongside decode as resources allow to avoid overwhelming the inference engine with a long sequence. But what if, after these optimizations, you still need more VRAM than a single GPU offers to store KV cache? You’ll need to parallelize inference across multiple GPUs. ## 5.4 Model Parallelism Every frontier LLM on the market today is too big to fit on a single GPU for batch inference. While GPUs have gotten bigger, so too have models, a trend that does not show signs of reversing. In FP8, loading a billion parameters of model weights takes roughly a gigabyte of VRAM. For a model like DeepSeek-V3.1, with 671 billion parameters, the model weights alone would cause a single B200 GPU to immediately throw an out-of-memory (OOM) error. It’s not enough to just barely squeeze the model weights into VRAM. On 4xB200 GPUs, with 720 GB of VRAM, you could theoretically load DeepSeek’s weights. But with no room left over for a KV cache, which often takes up 80 percent or more of the remaining VRAM after weights, four B200 GPUs would not be able to serve DeepSeek with any reasonable sequence length or batch size. Instead, a full node of eight B200 GPUs is needed to serve real production traffic on a model the size of DeepSeek. You can estimate the minimum number of GPUs required for a model by multiplying the precision, parameter count, and expected KV cache allocation together. ![Figure 5.11: After figuring out how much VRAM inference requires, round up to the next available instance size to determine minimum GPU count.](https://www.datocms-assets.com/104802/1788123437-inference-engineering-figure-5-11.png) _Figure 5.11: After figuring out how much VRAM inference requires, round up to the next available instance size to determine minimum GPU count._ In many cases, even for midsize models like GPT OSS, you want to use more than the minimum number of GPUs required to enable larger KV caches and unlock better per-user latency. However, all of this requires that inference scales efficiently from one GPU to multiple GPUs. The limitation in scaling parallel inference is the communication overhead between GPUs. Chapter 3 details the different interconnects between GPUs: NVLink and NVSwitch within nodes, InfiniBand between nodes. While NVLink and InfiniBand offer high bandwidth, they are a fraction of the speed of VRAM. With decode bound on memory bandwidth, multi-GPU inference needs to be carefully designed to avoid bottlenecks in inter-GPU communication. This field of study is called topology-aware parallelism. There are three primary forms of model parallelism in inference: - **Pipeline Parallelism (PP):** Splits the layers of the model across GPUs. - **Tensor Parallelism (TP):** Splits the tensors within each layer across GPUs. - **Expert Parallelism (EP):** Shards entire experts from MoE models across different GPUs. Each form of parallelism has its own tradeoffs: | Method | Mechanism | Drawback | | :----- | :---------------------------------------------------------------------- | :------------------------------------------------------------------------------ | | PP | Each GPU handles a stage of the forward and backward pass. | Not recommended due to poor latency and utilization from step-by-step pipeline. | | TP | Compute-heavy operations like matmuls are split across GPUs. | Requires synchronization across GPUs, not suitable for multi-node. | | EP | Each expert lives within a single GPU, making in-expert inference fast. | Requires routing between GPUs to reach multiple experts, better for throughput. | Tensor Parallelism is generally best for low-latency model inference within a single node, while Expert Parallelism improves throughput for MoE LLMs. Pipeline Parallelism is only used for multi-node inference. Additionally, data parallelism strategies like Context Parallelism split computation across devices. These strategies are rare in LLM inference but essential for video generation (section 6.6). ### 5.4.1 Tensor Parallelism for Lower Latency Tensor Parallelism should be your default strategy for multi-GPU model inference. It supports both dense models like Llama 405B and the MoE models that currently dominate the open model landscape. ![Figure 5.12: Tensor Parallelism splits weights across GPUs, effectively sharing VRAM resources to run large models fast.](https://www.datocms-assets.com/104802/1788123445-inference-engineering-figure-5-12.png) _Figure 5.12: Tensor Parallelism splits weights across GPUs, effectively sharing VRAM resources to run large models fast._ TP works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs. For each layer, the expense of reading from weights memory and executing matrix multiplication is shared across the GPUs. ![Figure 5.13: For Mixture of Experts models, each expert runs across multiple GPUs with Tensor Parallelism.](https://www.datocms-assets.com/104802/1788123450-inference-engineering-figure-5-13.png) _Figure 5.13: For Mixture of Experts models, each expert runs across multiple GPUs with Tensor Parallelism._ However, the results of each layer need to be communicated in an all-reduce fashion into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized. Increasing Tensor Parallelism improves TPS on a per-user basis (assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass, which is the case for most frontier models). ### 5.4.2 Expert Parallelism for Higher Throughput Expert Parallelism neatly divides experts across GPUs. In a model with 128 experts served in EP8 across eight GPUs, each GPU will host 16 full experts. ![Figure 5.14: Expert Parallelism runs each expert within a single GPU, with GPUs each hosting multiple experts.](https://www.datocms-assets.com/104802/1788123455-inference-engineering-figure-5-14.png) _Figure 5.14: Expert Parallelism runs each expert within a single GPU, with GPUs each hosting multiple experts._ EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens. Many deployments use a mix of TP and EP to achieve both benefits. ![Figure 5.15: This deployment uses TP for attention and EP for the sparse MoE layer.](https://www.datocms-assets.com/104802/1788123225-inference-engineering-figure-2-9.png) _Figure 5.15: This deployment uses TP for attention and EP for the sparse MoE layer._ Expert Parallelism requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model. Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer. Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth. ### 5.4.3 Multi-Node Inference If you’re serving a huge model at high precision, supporting multi-million-token input sequences, or just trying to run inference as fast as possible, you might need more than eight GPUs. ![Figure 5.16: InfiniBand enables multi-node inference across more than eight GPUs via high-bandwidth node-to-node interconnect.](https://www.datocms-assets.com/104802/1788123318-inference-engineering-figure-3-4.png) _Figure 5.16: InfiniBand enables multi-node inference across more than eight GPUs via high-bandwidth node-to-node interconnect._ GPUs are designed to work together across nodes, and multi-node training has been the standard for years to develop frontier models. But multi-node inference introduces new challenges: - **Infrastructure:** How do you reliably provision two or more interconnected GPU nodes and build abstractions across cloud providers (chapter 7)? - **Parallelism:** How do you effectively communicate over InfiniBand, which is much slower than NVLink? InfiniBand introduces a new wrinkle to topology-aware parallelism. Tensor Parallelism generally requires too much communication across GPUs to be a good fit for multi-node inference. Instead, you have two options that work well over InfiniBand: 1. For dense models, use Tensor Parallelism within each node and Pipeline Parallelism between nodes (e.g., TP8PP2). 2. For MoE models, you can also try Expert Parallelism (e.g., EP16) as it has a lower communication overhead than Tensor Parallelism. For MoE models, TP8PP2 will generally offer lower latency per user and EP16 will yield higher overall system throughput. Unless your model and KV cache are so large as to require multi-node inference, it probably isn’t the best use of the extra hardware. You’re often better off using the extra nodes for horizontal scaling across replicas, or for disaggregated serving. ## 5.5 Disaggregation Disaggregation combines three important ideas in inference engineering: 1. Prefill is a compute-bound process that determines your TTFT, while decode is a memory-bound process that determines your TPS. 2. Specialization improves performance in everything from kernel selection to inference engine parameter tuning. 3. You can effectively parallelize model serving over multiple GPUs, or even multiple nodes, if you can avoid bottlenecks from lower-bandwidth interconnects. When prefill and decode run on the same node under heavy traffic, they have a higher chance of interfering with one another. Ideally, prefill uses more compute resources, while decode uses more memory, and the two can co-exist efficiently. However, with larger batches and more compute-intensive optimizations, prefill and decode start competing for resources. ### 5.5.1 How Disaggregation Works Disaggregation, or disaggregated serving, is the idea of separating prefill and decode into separate engines on separate GPUs or nodes. ![Figure 5.17: Disaggregation assigns prefill workers to generate the first token and decode workers to generate subsequent tokens.](https://www.datocms-assets.com/104802/1788123460-inference-engineering-figure-5-17.png) _Figure 5.17: Disaggregation assigns prefill workers to generate the first token and decode workers to generate subsequent tokens._ Disaggregation turns LLM inference into a three-step process: 1. The prefill engine takes the input sequence and generates a KV cache while computing the first token. 2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine. 3. The decode engine computes all subsequent tokens. In conditional disaggregation, the request is first sent to the decode engine, which checks if the input sequence is already cached or is short enough to handle locally: 1. If it is, the decode engine handles prefill locally, skipping disaggregation. 2. If it is not, the decode engine transfers the request to the prefill engine for disaggregated serving. Conditional disaggregation is better for real-world traffic. Another benefit of disaggregation is that with separate prefill and decode engines, you can optimize each engine individually and the system as a whole. For example, the compute-bound prefill engine requires a lower TP than the memory-bound decode engine. ### 5.5.2 When to Use Disaggregation Disaggregation is very powerful but requires multiple GPUs and extra engineering work. You should reach for disaggregation only when: 1. You are serving a large volume of traffic, starting at one hundred million to one billion tokens per day depending on model size. 2. You are serving a larger model, at least a hundred billion parameters. 3. Your traffic is prefill-heavy with long input sequences. If either point one or two is not true, you’re likely wasting money on extra hardware for minimal performance gains. If point three is not true, you may be better off using the extra GPUs to scale replicas horizontally, as decode engines will be more efficient for short sequences or prefix cache hits. A great use case for disaggregation is serving a frontier LLM in a code editor, where many developers are simultaneously passing in large and varied chunks of code as context. Tons of tokens, mostly prefill, on a trillion-parameter LLM is the textbook workload for disaggregation. ### 5.5.3 Dynamic Disaggregation with NVIDIA Dynamo Dynamo provides production-ready support for disaggregation, with flexibility to handle heterogeneous real-world traffic. Dynamo provides developer tools and pre-built optimizations to enable disaggregation: - A prefill queue to hold requests when all prefill engines are saturated. - Robust support for conditional disaggregation, with prefill routing based on configurable thresholds for ISL after prefix cache and prefill queue size. - Efficient NIXL-based KV transfer from prefill to decode engines with a kernel to transpose KV blocks between layouts when the engines have different TP configurations. Combined, these features enable dynamic disaggregation, where the number of prefill and decode engines is configurable at runtime and can be adjusted over time to match the changing nature of incoming traffic. Disaggregation does not need to be a one-to-one ratio between prefill and decode engines. While it’s simple to explain disaggregation in terms of a single prefill engine and a single decode engine, real systems have multiple of each. The number of prefill and decode engines is written as xPyD, for example, 5P3D means five prefill and three decode engines working together to serve a single model deployment. As systems grow more complicated, more potential bottlenecks appear. With disaggregation, the new bottleneck is prefill queue size. It’s important to not let the queue grow too large, both by setting a reasonable threshold for local prefill on the decode engine and by reconfiguring xPyD at runtime to allocate more resources to prefill if needed. The other potential bottleneck in disaggregation is running out of KV cache on the decode engines under high load. Increase KV cache availability with quantization and KV cache offloading.