5.1 Quantization

Quantization improves latency (both TTFT and TPS), increases system throughput, and opens up headroom for other optimizations like disaggregation, speculation, and prefix caching to be even more effective. But when it goes wrong, quantization can materially reduce the model’s output quality.

Models are trained with weights, activations, and other components represented in a certain native number format. Usually, this is BF16 or FP16, though 8-bit and 4-bit native precisions are becoming more popular in training.

Post-training quantization works by changing those model weights and other values from their native number format to a lower-precision format. Cutting precision in half improves performance in both phases of inference:

• Prefill: Compute-bound prefill now runs on lower-precision Tensor Cores with twice the FLOPS.
• Decode: Memory-bound decode now loads half as much data per value, effectively doubling memory bandwidth.

Working with quantized data does introduce overhead, so it’s not linearly twice as fast to go from 16 to 8 bits. In practice, quantization down a single level of precision generally offers 30 to 50 percent better performance for LLMs.

The catch with quantization is that it runs the risk of reducing the model’s output quality. Quantization has the potential to introduce precision errors throughout the calculations that power inference.

Precision errors compound over time. Consider what happens when you square and cube different precisions of Pi:

Most of the work in quantization is around both preventing precision errors and minimizing their impact on the final model output.

5.1.1 Number Formats

Quantization introduces a new collection of essential terms and abbreviations. The most important ones to know are the common number formats:

The largest number format, FP64 or “double precision,” is only used for high-precision scientific computing, not AI training or inference. FP32 is sometimes used for training, but almost never for inference. FP6 is more experimental at the time of publication, though AMD GPUs are rapidly adopting the format.

That leaves 16, 8, and 4-bit precisions as the primary formats for inference. Number formats have a:

• Precision: The number of bits used to express a single value in the format. For example, FP16 uses 16 bits.
• Type: Whether these bits are interpreted to represent an integer (no decimal) or a floating-point number (a decimal).
• Scale factor: A multiplier used to map values from a low-precision format back to the higher-precision format.

Together, these attributes determine the two factors behind how well a number format represents values used in inference:

• Dynamic range: The difference between the lowest and highest value that can be represented in the format.
• Granularity: The number of parameters or other values that are quantized along a single scale factor.

Dynamic range is essential to low-precision inference without quality loss. 16 bits can represent 65,536 distinct values, while 8 bits can only represent 256 different values. The dynamic range is the distribution of these values – the difference between the smallest and largest available value.

Dynamic range explains why floating-point formats are better than integer formats for inference. Floating-point formats have three properties:

• Sign: A single bit that represents whether the number is positive or negative.
• Exponent: A set of bits that, taken together, represent an exponent factor.
• Mantissa: A set of bits that, taken together, represent the base value multiplied by two to the exponent.

An FP8 number in a E4M3 data format means it has a 4-bit exponent and a 3-bit mantissa, with the remaining bit for the sign. Integer formats only have sign and value bits.

The exponent in floating-point numbers gives it a higher dynamic range, meaning it can better express very large and very small numbers. This is important because outlier values are significant in inference, and floating-point number formats better represent outliers after quantization.

Within floating-point formats, there are multiple options at each precision, like FP4, MXFP4, and NVFP4. These formats differ in granularity, or the number of values that are quantized by a single scale factor.

Quantization can be applied at the:

• Tensor level: Calculate a single scale factor for the entire QKV tensor.
• Channel level: Calculate a different scale factor for each feature vector within the tensor.
• Block level: Within each feature vector, divide the vector into blocks of N values and calculate a scale factor for each block.

More granular quantization has a lower chance of smoothing over outliers, preserving quality. However, more granularity introduces more overhead for storing and applying scale factors.

MXFP8 and MXFP4, the new number formats supported by Blackwell, are “microscaling” formats that compute a blockwise scale factor on every 32 parameters, reducing the impact of these number formats’ lower dynamic range.

NVFP4, a 4-bit format by NVIDIA, offers even higher granularity than the MX formats with a block size of 16 and a secondary 32-bit global scale factor to further combat the quality loss that 4-bit formats introduce.

The tradeoff to a microscaling format is that the small-block scale factor must also be stored in memory, slightly reducing the performance gains from quantization. Additionally, both the tensor and block scale factors need to be applied, introducing a bit of compute overhead. Blackwell GPUs offset this overhead via scale factor application in Tensor Cores.

While the focus of this book is on inference engineering in the datacenter, quantization is an essential topic for local and edge inference, especially for large models. GGUF, a binary format for storing models, is the most popular choice for distributing highly quantized models on Hugging Face, with individual researchers and companies squeezing huge models like DeepSeek onto consumer hardware like Apple computers.

These quantization strategies combat quality loss through dynamic quantization, where certain layers or other components of the model are left in their original precision, while others are quantized to integers with as little as one bit of precision. Dynamic formats represent their average precision, which is why you might see something like fine-tuning lab Unsloth’s popular 1.58-bit quantization.

While these dynamic quantizations are impressive feats of engineering and are great for local inference, inference engineers working on production systems should stick with floating point number formats – integer formats are not suitable for quality-sensitive workloads due to their lack of dynamic range.

Instead, 8-bit floating-point formats (FP8, MXFP8) are generally the sweet spot for improving performance without sacrificing quality. FP4 is promising, especially the NVFP4 format which introduces a higher level of granularity for improved accuracy, but FP8 and MXFP8 provide the most flexibility, especially when quantizing the KV cache.

5.1.2 Quantization Approaches

The more parameters a model has, the less sensitive it is to quantization as each individual parameter is less important. However, even for very large models, it is essential to quantize carefully.

Quantization can happen during or after training:

• Quantization-aware training: Training weights and computing scale factors together to ensure that the final converged weights are accurate at a given precision.
• Post-training quantization: Converting finished model weights to a new precision by computing scale factors and preserving accuracy via calibration.

While some labs release models created with quantization-aware training, like GPT-OSS in MXFP4 and Kimi K2 Thinking in INT4, inference engineers working with open models only have the ability to perform post-training quantization as they are working with finished weights.

A leading tool for post-training quantization is NVIDIA TensorRT Model Optimizer (ModelOpt), an open-source library that also supports pruning, distillation, and sparsity. ModelOpt outputs are compatible with all inference engines (vLLM, SGLang, TensorRT-LLM).

After picking a precision, there are two decisions to make before doing post-training quantization:

1. What parts of the model (weights, activations, KV cache, attention) should be quantized?
2. What number format offers the appropriate dynamic range and granularity?

These decisions turn quantization from a binary choice into a spectrum of tradeoffs around performance and quality.

Components of a model have varying sensitivity to quantization. Reducing the precision of more sensitive components runs a higher risk of quality degradation. From least to most sensitive:

1. Weights: Specifically the linear layers are least sensitive to quantization.
2. Activations: The intermediate output of activation functions are only somewhat sensitive to quantization. Note that the activation functions themselves are rarely quantized as they are such a tiny fraction of the model’s weights.
3. KV cache: The cached values from the attention calculation are moderately sensitive to quantization.
4. Attention: The attention layers of a model are highly sensitive to quantization, especially equations like softmax.

Within each of these components, you can get more selective about quantization.

Even in linear layers and activations, which are generally the least sensitive to quantization thanks to their size, early and late layers like the input and output layer of the neural network may be left in their original precision as these layers are more sensitive.

While quantizing weights and activations helps performance directly, KV cache quantization gives an additional boost to techniques like prefix caching and disaggregation. The KV cache is a valuable resource. Quantizing it allows inference engines to store more of it in memory and read it more quickly.

However, the KV cache for each token is used by each subsequent token. This means precision errors introduced by quantization can compound from token to token.

Compounding errors is exactly the reason why attention layers are the riskiest to quantize. Not only is attention very sensitive to dynamic range, but each attention calculation relies on the results of each previous attention calculation. Over a sequence of thousands of tokens, these errors accumulate quickly.

All but the most aggressive quantization schemes run functions like softmax in their original precision.

A moderate approach to low-precision inference uses a format like FP8 with high dynamic range – if possible, a microscaling format like MXFP8 – to carefully quantize select linear layers, activations, and often KV cache values. Even with these high dynamic range formats, components of the attention layer are rarely quantized.

5.1.3 Measuring Quality Impact

The standard for production-ready quantization is zero perceptible quality loss. After quantizing a model, it’s essential to thoroughly test its output quality versus the original precision.

There are three methods for checking model quality after quantization:

1. Perplexity: Calculate the perplexity score for the quantized model and compare with the original.
2. Intelligence benchmarks: Run a standard intelligence benchmark like MMLU or SWE-bench and compare to original scores.
3. Custom evals: Run a product-specific evaluation suite on the quantized model and compare to the original weights.

In every case, you’re looking for a difference in scores that’s indistinguishable from noise. LLMs are non-deterministic, so scores vary slightly from run to run.

The simplest check on quality is perplexity. Rather than asking the model to generate output, perplexity gives the model expected output sequences and calculates the likelihood of the model predicting those tokens.

A higher perplexity means a model is more “surprised” by the sequences – not what you want from a model that’s supposed to predict tokens. After quantization, you’re looking for a minimal increase in perplexity.

A more comprehensive quality check relies on a public intelligence benchmark or, better yet, a domain-specific eval that matches your expected real-world usage. On evals, you’re looking for a minimal reduction in the quality score.

The best way to get the full picture of the impact of quantization is to run all three types of checks and make apples-to-apples comparisons to the original model weights.

Remember that quantization is a scale, not a binary decision. You can still get some performance improvements with lower risk of quality loss by quantizing to FP8 instead of FP4, or by quantizing fewer components of the model, like weights-only quantization.

If you’re working in a highly sensitive domain and can’t risk model quality, no worries: every other technique in this chapter is lossless in terms of quality.