One of the coolest parts about working in inference engineering is that unlike many industries where new academic research takes years or decades to be adopted by industry, techniques from new papers are live in production within months or even weeks.
There is a gap to cross between research and production, and some of the most visible inference engineering work in the industry comes from bridging that gap.
A core principle of inference engineering is that the more constraints you can introduce in your inference system, the better performance you’ll achieve. This principle continues to apply throughout this chapter, with techniques like disaggregation, which allows you to constrain individual engines to prefill and decode.
With these model performance techniques, there’s a new principle to keep in mind: the more traffic you have, the more performance optimizations you can make (while keeping unit economics reasonable). Higher model parallelism across more GPUs, KV-aware routing, and dynamic disaggregation only make sense when you have a large number of GPUs, often multiple nodes, serving the same model with vertical scale and horizontal replication.
Real-world traffic defies constraints. But with volume, you can adapt your systems over time to match the changing nature of usage. Tuning the parameters of inference engines, speculation algorithms, and model servers isn’t a one-time task. Instead, either through iterative deployments or dynamic runtime adjustments, you can continuously improve the performance of your inference system.
Finding the right combination of techniques and configurations takes patient experimentation. I remember an internal hackathon during which one of Baseten’s inference engineers was working on an autocomplete model for code and ended up trying 77 different configurations via a handwritten script before finding a non-obvious solution that doubled TPS for a customer’s model.
To make inference optimization even more complex, sometimes techniques are symbiotic and sometimes they are incompatible. For example, quantizing the KV cache alleviates a bottleneck in disaggregation, but increasing batch sizing reduces the compute available for speculation. An inference engineer’s goal is always to create a balanced set of optimizations that delivers more than the sum of its parts.
This chapter introduces five key categories of applied research for inference acceleration: quantization, speculation, caching, parallelism, and disaggregation. In each section, pay special attention to the recommended circumstances for using each technique and the potential bottlenecks or tradeoffs each introduces.
- 5.1 Quantization: Lowering precision to FP8, FP4, and beyond to cut memory traffic and unlock compute without losing quality.
- 5.2 Speculative Decoding: Draft tokens, validation, and acceptance rate: generating more than one token per forward pass.
- 5.3 Caching: KV cache mechanics, prefix caching across requests, and cache-aware routing between replicas.
- 5.4 Model Parallelism: Tensor, pipeline, expert, and data parallelism for models too large to serve on a single GPU.
- 5.5 Disaggregation: Separating prefill and decode onto independently scaling workers, and when that pays off.