# Chapter 4: Software _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) NVIDIA’s market dominance in the inference space is in no small part due to the robust and mature software ecosystem around its hardware. Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most yearly, with two-year release cycles being more common. But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other prerelease version of each of your software dependencies just to get support for the new model. Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. While hardware centers on NVIDIA and a few competitors, there are countless companies building software at various levels of the inference stack. For inference engineers, these are some key players: - **NVIDIA:** Invests heavily in its own sometimes-proprietary software ecosystem, from CUDA up to Dynamo. - **Hugging Face:** Maintains a model registry for all open models plus `transformers` and `diffusers`. - **The Linux Foundation:** Maintains hardware-agnostic projects like PyTorch and vLLM. - **LMSYS Org:** Develops essential tools for inference and evaluation, most notably SGLang. There are thousands more companies, universities, and research institutions making essential open-source contributions to inference. The software space is too big and too fast to exhaustively document in any book, much less in a single chapter. Instead, this chapter presents foundational technologies with long-term relevance. Throughout the chapter, technologies are presented with increasing levels of abstraction: - **CUDA:** Direct communication to the GPU for explicit control over computations and memory (section 4.1). - **Deep learning frameworks:** Abstractions over CUDA for training, exporting, and running neural networks in Python (section 4.2). - **Inference engines:** Highly configurable PyTorch-backed inference for common architectures (section 4.3). - **NVIDIA Dynamo:** Sits on top of inference engines to power large-scale deployments (section 4.4). Most inference engineering today happens at the higher levels of abstraction, configuring and deploying inference engines and orchestrating inference across multiple GPUs. No matter what level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work. ## 4.1 CUDA CUDA is how you write code to run on NVIDIA GPUs. More formally, CUDA is NVIDIA’s proprietary computing platform and programming model for executing parallel tasks on GPUs. Both “platform” and “programming model” are broad definitions; it’s easier to look at CUDA in its component parts: - **CUDA kernel:** A user-defined function that executes parallelized code on the GPU. - **CUDA graph:** A directed acyclic graph (DAG) of kernels and other GPU operations for optimizing repeated workflows. - **CUDA driver:** A low-level interface between the application and the GPU hardware to manage memory and execution. - **CUDA runtime:** A developer-facing API for launching kernels and managing memory. CUDA – which stands for Compute Unified Device Architecture, though the acronym is rarely expanded today – is the foundation for the entire generative AI ecosystem on NVIDIA GPUs. CUDA is not a programming language. Instead, CUDA programs are specified in a programming language, most often C++, then compiled into separate CPU and GPU code by a compiler like `nvcc`. A kernel is just a function that does some parallel computation. Whenever you see the phrase “CUDA kernel” you can replace it with “a piece of code written for NVIDIA GPUs.” For a “Hello, World!” kernel, consider these six lines of C++ code which take in an array of length `n` and double each element in the array. ![Figure 4.1: Example CUDA kernel doubles each value in an array.](https://www.datocms-assets.com/104802/1788123334-inference-engineering-figure-4-1.png) _Figure 4.1: Example CUDA kernel doubles each value in an array._ Ordinarily, on a CPU, this function would run in linear time, with each element in the array being doubled sequentially. But on a GPU, thousands of elements can be processed simultaneously, making this function much more efficient. Writing CUDA kernels shifts inference engineering from thinking about algorithms to thinking about implementations. For example, the traditional attention algorithm that is central to generative AI can be expressed in a few dozen lines of code. However, FlashAttention, which is the same mathematical operation, takes tens of thousands of lines of code to implement the algorithm in a more memory-efficient manner for a specific GPU. ### 4.1.1 CUDA Kernels for Inference Writing CUDA kernels does not mean building from scratch. The prior art to build upon predates CUDA by decades. BLAS (Basic Linear Algebra Subprograms), first implemented for Fortran in the 1970s, is a specification for common linear algebra operations from dot products to matrix multiplication. cuBLAS, the CUDA implementation for BLAS, brings this specification to CUDA in the form of pre-built kernels for essential linear algebra operations. Similarly, cuDNN (CUDA Deep Neural Network) provides primitives for neural networks. Within BLAS, the most frequently used operation in inference is GEMM (General Matrix-Matrix Multiplication). Every linear layer in a model uses matrix multiplication, and cuBLAS provides a strong starting point. But you aren’t limited to the cuBLAS implementation. As operations like GEMM are so essential for inference, you may need more fine-grained control. You might write different GEMM kernels for matrices of different shapes or to better run on specific GPU architectures. CUTLASS is a template library that provides building blocks for writing high-performance kernels. For example, FlashAttention 3 uses CUTLASS. CuTe, another template library, introduces abstractions for tiled tensor operations for recent architectures. With tools like CUTLASS and CuTe, kernels can be written at a higher level of abstraction while retaining strong performance. Another source for kernels is FlashInfer, a library with high-performance implementations of kernels for LLM inference, including many optimized attention kernels and fused sampling functions. ### 4.1.2 CUDA Kernel Selection Most inference engineers will never need to write their own kernels. However, kernel selection – choosing the best kernel from a range of options – is an important part of inference optimization. Kernel implementations are highly specialized. CUDA exposes low-level APIs for memory management and parallel computing, and kernel engineers make implementation decisions based on the exact specifications of the hardware they’re writing for. It’s important to understand how closely tied kernel implementations are to specific hardware details. Kernels often have hard-coded values based on the memory bandwidth or the number or layout of Tensor Cores on a given GPU. A kernel written for an H100 will likely fail to take advantage of the architecture and extra memory of a B200, while a kernel written for that B200 could be backwards incompatible with the previous-generation Hopper architecture. With each generation of GPUs, porting handwritten kernels to run optimally on the new architecture takes substantial engineering work. Most kernel selection is automatic. Deep learning frameworks and inference engines have pre-configured kernels for various architectures, while PyTorch and TensorRT-LLM include automatic kernel selection in their compilation steps. However, you might manually choose a few kernels for essential algorithms to speed up inference. For example, most production-ready GEMM kernels come from cuBLAS. But when the DeepSeek lab released their updated version of DeepSeek-V3, they also released DeepGEMM, which provides more efficient GEMM kernels for running matrix multiplication in FP8 on the Hopper GPU architecture. Manual kernel selection lets you insert a DeepGEMM kernel as a plugin to speed up a specific step in inference, like multiplying two matrices of precise dimensions. Just be careful to ensure compatibility. For example, if you were to upgrade to a B200 GPU, you’d have to either swap the kernel back, wait for Blackwell support in DeepGEMM (now supported as of the time of publication), or port the kernel yourself. ### 4.1.3 Reducing Memory Accesses with Kernel Fusion Running two different kernels back-to-back on the same data results in wasted reads from and writes to memory. As a simple example, imagine two kernels: `multiply_by_2` and `multiply_by_3`. If these kernels were run back-to-back, the sequence would be: 1. Read the input vector `[1, 2, 3]` from memory. 2. Run `multiply_by_2` on the vector. 3. Save the output vector `[2, 4, 6]` to memory. 4. Read the new input vector `[2, 4, 6]` from memory. 5. Run `multiply_by_3` on the vector. 6. Save the final output vector `[6, 12, 18]` to memory. The inefficiency is clear: steps three and four comprise an unnecessary round trip to memory. During decode, the bandwidth-bound phase of LLM inference, an inference engine can’t afford unnecessary reads from or writes to memory. Kernel fusion is the process of taking two or more kernels and re-implementing them into a single kernel that handles both operations. In this example, the fused kernel would be `multiply_by_6` and the new order of operations would be: 1. Read the input vector `[1, 2, 3]` from memory. 2. Run `multiply_by_6` on the vector. 3. Save the final output vector `[6, 12, 18]` to memory. In practice, kernel fusion is far more complicated – functions are more complex, and data overlap isn’t as clean. But there are common patterns in kernel fusion for inference, like combining matrix multiplication, bias adding, and activation. ![Figure 4.2: Kernel fusion reduces reads and writes between memory and compute within a GPU.](https://www.datocms-assets.com/104802/1788123340-inference-engineering-figure-4-2.png) _Figure 4.2: Kernel fusion reduces reads and writes between memory and compute within a GPU._ Kernel fusion can be an automatic or a manual process. Compilers can identify straightforward fusion opportunities and automatically create fused kernels. But more sophisticated algorithms, like FlashAttention, require handwritten fused kernels, which are used via plugins during inference. ## 4.2 Deep Learning Frameworks and Libraries Deep learning frameworks and libraries are the bridge between working directly in CUDA and using off-the-shelf inference engines like vLLM. These libraries are used in both training and inference. Over the past few years, PyTorch has emerged as the clear leader at this level of the stack. There are two other frameworks that, for concision, I will only mention briefly: - **TensorFlow:** An end-to-end machine learning platform officially supported by Google, TensorFlow was prominent in the 2010s ML era but has fallen out of favor today. - **JAX:** A research project unofficially associated with Google, JAX presents a simpler interface without as many legacy features and operations. However, as the documentation warns, expect sharp edges. The remainder of this section focuses on PyTorch and the technologies around and above it in the stack. ### 4.2.1 PyTorch PyTorch is a Python package for describing tensor operations. Originally created at Meta and now a part of the Linux Foundation, PyTorch is the industry standard technology underlying both training and inference for generative AI models. Personally, I’ve been a Python programmer for my entire career, and I find writing low-level C++ difficult. With PyTorch, I can write highly performant inference code in Python for both CPUs and GPUs, but I always have the option to dip down into CUDA when necessary by plugging in specific kernels. PyTorch can train any kind of neural network. The PyTorch documentation shows a basic neural network example: ![Figure 4.3: A basic neural network in PyTorch, adapted from the PyTorch documentation.](https://www.datocms-assets.com/104802/1788123348-inference-engineering-figure-4-3.png) _Figure 4.3: A basic neural network in PyTorch, adapted from the PyTorch documentation._ While this is a very simple example, you may recognize the linear layers and ReLU activation functions discussed in chapter 2 as key pieces of neural networks. PyTorch automatically computes gradients for any differentiable function via its `autograd` module. This is what makes PyTorch so powerful for training – you define a computation graph, and you get a gradient to train against. But PyTorch is special because it isn’t just great for training, it’s also powerful for inference. PyTorch balances built-in functions and automatic performance optimizations with manual control where you need it. The step that transforms a model from training to inference is compilation. PyTorch compilation (`torch.compile`) targets a specific GPU and performs automatic kernel selection and kernel fusion to ensure optimal performance. `torch.compile` can’t fuse plugin kernels like DeepGEMM, FlashAttention, or custom kernels. This limits its utility for LLM inference, where most kernels are custom. However, PyTorch compilation is useful for optimizing less common model architectures and compiling long sequences of lightweight kernels. When you are optimizing a model that has a custom or rare architecture, you may have to rewrite functions to be more abstract – especially with respect to Python-specific language features – for compilation to succeed. PyTorch alone is a powerful and flexible tool for building high-performance inference services. But there is a rich ecosystem on top of PyTorch that makes it faster to implement, compile, and execute optimized code for common model architectures. ### 4.2.2 Model File Formats The dominant file format for serializing model weights is safetensors, created by Hugging Face. Safetensors is a replacement for generic formats like bin designed specifically for holding model weights. The safety in safetensors comes from the fact that unlike a general format that can execute arbitrary Python code during deserialization, safetensors only hold tensor data, not executable code. Generative AI models have hundreds of gigabytes of weights. These weights are split across dozens of safetensors files. The safetensors format uses memory mapping to ensure that the files are loadable without allocating the full memory, which makes loading model weights faster and safer. Another leading format, ONNX (Open Neural Network Exchange), stores weights along with an execution graph for the model. Where the safetensors format separates the weights from the architecture, ONNX bundles them together. ONNX files are highly portable. With deep integration into PyTorch and support for multiple hardware options, ONNX is a great alternative to safetensors for when you want to store model graphs, not just weights. ### 4.2.3 ONNX Runtime and TensorRT ONNX Runtime and TensorRT are high-performance inference runtimes. PyTorch models can be exported to the ONNX format, which ONNX Runtime can execute directly or TensorRT can compile into a highly optimized engine. | ONNX Runtime | TensorRT | | :------------------------------------------------ | :------------------------------------------------------ | | Open source, associated with the Linux Foundation | Mix of proprietary and open components, built by NVIDIA | | First-class exporter in the PyTorch ecosystem | Integrated with PyTorch via Torch-TensorRT | | Supports many types of GPUs | NVIDIA GPUs only | ONNX Runtime is an open community standard, while TensorRT is specific to NVIDIA GPUs. The export process looks somewhat like Torch compilation. However, these standards do not support every data structure, type, and operation within PyTorch. The export process can identify these issues in PyTorch code, but this gets tricky with more complex models. An example is DeepSeek V3, which introduces Multi-Latent Attention (MLA). MLA, as implemented in PyTorch, is difficult to export, but the transformers architecture overall is simple enough that hand-fusing kernels is feasible. Today, it’s increasingly popular to skip directly from PyTorch to an inference engine like vLLM or TensorRT-LLM for models that these engines support, bypassing the intermediate representation step and exporting weights only as safetensors. ONNX Runtime and TensorRT are still widely used – TensorRT especially for its strong out-of-the-box runtime for image and video models – but the industry is bifurcating between the control of handwritten PyTorch code or the convenience of prebuilt inference engines. ### 4.2.4 Transformers and Diffusers The `transformers` and `diffusers` libraries by Hugging Face are built on PyTorch but are not designed to run large-scale production inference. Instead, these libraries offer reference implementations of models for inference engineers to learn from and adapt. While these libraries are toolboxes for tinkering, they do include essential information about models and useful utilities for building model inference servers. The `config.json` file that ships with models implemented for `transformers` and `diffusers` contains essential information, and the libraries’ utilities for Hugging Face operations like downloading model weights are widely used. You’ll find `transformers` or `diffusers` sample code in the model card for most popular open models on Hugging Face. This sample code is great for understanding the exact input and output spec of a model or for running local inference and notebooks. But for production, you’ll want to either write and compile PyTorch code directly or use a production-ready inference engine. ## 4.3 Inference Engines There are three competitive inference engines on the market: vLLM, SGLang, and TensorRT-LLM. These frameworks offer good out-of-the-box performance for LLMs and other modalities with similar architectures (chapter 6). In late 2025, vLLM and SGLang also began supporting some image and video generation models via vLLM Omni and SGLang Diffusion, respectively. TensorRT-LLM does not support image or video generation models. Inference engineers can also use TensorRT or PyTorch directly to run these models (section 6.5). Inference engines are powerful because they are configurable. Working with pre-optimized components at a higher level of abstraction, inference engineers can spend their time testing combinations of techniques rather than repeating routine implementations. At a very high level, vLLM and SGLang are more general tools that are easier to adopt and have day zero support for more models, while TensorRT-LLM has a steeper learning curve but usually achieves the best performance. | Engine | vLLM | SGLang | TensorRT-LLM | | :------------ | :--------- | :---------- | :----------- | | Performance | Good | Good | Best | | Ease of use | Easy | Easy | Hard | | Model support | Most | Most | Some | | Hardware | GPU, TPU | NVIDIA, AMD | NVIDIA only | | License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Each framework runs out of the box with core features like continuous batching and supports the main performance optimization techniques – post-training quantization, speculative decoding, prefix caching, parallelism, disaggregation. At Baseten, we use all three frameworks, though we use TensorRT-LLM the most. Inference engineers should be familiar with all three and select on a deployment-by-deployment basis. ### 4.3.1 vLLM vLLM has the largest market share among inference engines. GitHub stars are a rough measure for popularity, but at the time of publication vLLM has twice as many stars as SGLang and TensorRT-LLM combined. First released in the summer of 2023, vLLM is the oldest of these inference engines by a few months. Originally created at UC Berkeley, vLLM is now hosted by The PyTorch Project within The Linux Foundation. vLLM’s best selling point is its broad support. It supports the most hardware options – NVIDIA, AMD, and Intel GPUs along with Google TPUs – as well as the most models and architectures. Just about every open LLM out there integrates with vLLM from day zero. vLLM also supports multimodal inference via vLLM Omni, which extends the engine to support image, audio, and video inputs and outputs. One of the core principles of inference engineering is that the more constraints you can introduce, the better performance you can achieve. vLLM’s broad platform can achieve impressive performance results when properly configured, but in my experience it falls short of the highest-end performance possible with narrow frameworks like TensorRT-LLM. vLLM’s developer experience is built around the `vllm serve` command, with server configuration passed in as flags. ![Figure 4.4: vLLM inference example on eight GPUs.](https://www.datocms-assets.com/104802/1788123353-inference-engineering-figure-4-4.png) _Figure 4.4: vLLM inference example on eight GPUs._ vLLM is pip-installable and provides official Docker images with pre-bundled dependencies and support for various hardware architectures. You should use vLLM when: - You want to quickly stand up a model server that will offer solid performance out of the box for almost any open model. - You want to run an “Omni” model with multiple input and output modalities. - You are using a smaller GPU or older architecture where TensorRT-LLM offers few performance benefits. ### 4.3.2 SGLang SGLang is the other major community-driven fast inference framework. First released in December 2023, SGLang has risen to prominence alongside Chinese open models like DeepSeek and Qwen and is the engine of choice for inference at xAI. SGLang’s unique angle on the problem of model serving is expressed in its developer experience, which pairs a fast backend runtime with a flexible frontend language. In practice, that means you can choose individual components of your engine for deep customization without needing to rewrite everything else from scratch. SGLang supports both NVIDIA and AMD GPUs, and has strong day-zero support for a wide range of models. SGLang works closely with labs like DeepSeek, Qwen, Kimi, and Z AI to release optimized implementations of new architectural features like DeepSeek’s Multi-Latent Attention. SGLang has invested heavily in supporting large-scale deployments of MoE LLMs, specifically multi-node deployments on systems like GB200 NVL72 for high throughput. These systems offer extremely cost-efficient inference for large models with significant traffic. SGLang’s developer experience is built around the `sglang.launch_server` command, with server configuration passed in as flags. ![Figure 4.5: SGLang inference example on eight GPUs.](https://www.datocms-assets.com/104802/1788123358-inference-engineering-figure-4-5.png) _Figure 4.5: SGLang inference example on eight GPUs._ SGLang also supports image and video generation model inference via SGLang Diffusion. SGLang Diffusion introduces a pipeline abstraction which orchestrates a number of stages. This flexible approach maps closely to the architecture of image and video generation models. For performance, SGLang Diffusion adds support for various diffusion-specific parallelism methods and re-uses the scheduler and optimized kernels from the main SGLang package. You should use SGLang when: - You want excellent out-of-the-box throughput with decent latency on large MoE models like DeepSeek and Kimi. - You want the inference engine experience for image and video generation models. - You want control and customization and are excited to participate in the SGLang community. ### 4.3.3 TensorRT-LLM TensorRT-LLM is NVIDIA’s open-source inference engine. Of the three main options, TensorRT-LLM offers the highest performance and the most flexibility to expert users. A note on naming: There are two major versions of TensorRT-LLM. Only the older version is actually related to TensorRT: - **TensorRT-LLM V0 (0.X.Y):** Major versions starting with zero are a plugin for NVIDIA TensorRT. - **TensorRT-LLM V1 (1.X.Y):** Major versions starting with one are a standalone package based on PyTorch with no dependency on TensorRT. Originally, TensorRT-LLM built a TensorRT engine for serving language models. With the modern PyTorch-based version, TensorRT-LLM bypasses the intermediate representation of TensorRT and uses PyTorch directly. TensorRT-LLM V1 was released in the summer of 2025. Deployments of the previous major version are still common – always be sure to check which version you are using. TensorRT-LLM achieves the best performance in large part because it has access to kernels written by NVIDIA engineers, including some closed-source kernels. These handwritten and manually fused kernels offer excellent support for the latest hardware architectures like Hopper and Blackwell and for NVIDIA-specific number formats like NVFP4. TensorRT-LLM offers a robust implementation of in-flight batching (token-level continuous batching), which helps with throughput. It also supports just about every model performance optimization setting you could ask for, including quantization, speculation algorithms, prefix caching, chunked prefill, flexible parallelism, and disaggregation. With V1, TensorRT-LLM introduces a developer experience that looks a lot like vLLM and SGLang. However, in addition to flag arguments on the `trtllm-serve` command, it expects a `config.yaml` file for deeper customization. ![Figure 4.6: TensorRT-LLM inference example on eight GPUs.](https://www.datocms-assets.com/104802/1788123363-inference-engineering-figure-4-6.png) _Figure 4.6: TensorRT-LLM inference example on eight GPUs._ The best way to install TensorRT-LLM is by running it via one of NVIDIA’s official Docker containers Use TensorRT-LLM when: - You are running a well-supported model architecture on a Hopper or later GPU. - You are willing to do extra engineering work to get the best possible performance. - Optionally, you are planning to use NVIDIA Dynamo for serving and want the most deeply integrated engine. ## 4.4 NVIDIA Dynamo NVIDIA Dynamo is a distributed system for model serving first announced at NVIDIA GTC in March 2025. Dynamo works with every inference engine – vLLM, SGLang, and TensorRT-LLM – as backends, with Dynamo itself providing an orchestration layer for large-scale deployments. Dynamo provides support for essential model performance techniques: - **KV cache re-use:** Retaining KV information between requests and routing requests based on prefix match. - **Disaggregation:** Separating prefill and decode onto individually optimized engines with independent scaling. - **Multi-node parallelism:** Optionally using two or more nodes of GPUs in a single replica for a model, usually with Expert Parallelism. Each of these techniques will be detailed in chapter 5. As with the inference engines, there is a lot of work for inference engineers to do to configure Dynamo for their use case and achieve maximum performance. Dynamo’s thoughtful abstractions for distributed KV routing, disaggregation, and multi-node model parallelism provide high-performance aggregation of information during runtime, allowing for real-time adjustments to configuration as traffic fluctuates. For example, you can automatically scale up and down prefill and decode workers with an SLA-based planner operating on user-defined TTFT and TPS constraints. As a general principle, the more scale you have, the more tools and techniques there are available to you for inference optimization. Dynamo is built for scale: big models, big traffic. It excels at serving foundation models like the trillion-parameter Kimi family to a large number of concurrent users. For smaller models, Dynamo can still offer moderate performance improvements on large-scale deployments. If you’re building an inference API for a built-from scratch foundation model or serving an open model in a high-usage product, Dynamo is a great choice. But many deployments don’t need the additional complexity of Dynamo. Unless you’re operating with enough volume for disaggregation and KV-aware routing to matter, Dynamo will be unnecessary work and excess overhead. In these cases, you can use inference engines directly. Dynamo is the newest project covered in this chapter, and features are still being built out. Dynamo is open source under the Apache 2.0 license. The community around Dynamo is active, and the project welcomes contributions with a public CI and support from NVIDIA engineers. ## 4.5 Performance Benchmarking and Load Testing Benchmarking is an essential part of model performance optimization. Without precise, accurate performance benchmarks, there’s no way of knowing if your optimizations are actually working. A high-quality benchmark simulates real life as closely as possible. The best benchmark is to shadow real-world production traffic onto the system you are testing. Shadowing is the process of copying incoming requests onto the test system so that you can benchmark its performance without affecting the original request. If you can’t shadow real usage, you’ll need to simulate it. LLM performance is affected by a number of factors. When simulating traffic, you need to match your expected production workload on multiple dimensions: - **Sequence lengths:** Time to first token and memory usage rely on the input sequence length (ISL) and output sequence length (OSL), meaning the number of tokens in the prompt and response. - **Volume and pattern of traffic:** Batching and server load depend on the number of concurrent requests. Jitter traffic to mimic real usage. - **Request contents:** The actual prompt within each request affects performance factors like cache hit rate and draft token acceptance. - **Input parameters:** Settings like temperature and reasoning effort that affect inference should be set to their anticipated production values. Remember, optimization is about tradeoffs and constraints. If you’re maximizing benchmark performance against bad inputs, performance in production won’t match expectations. ### 4.5.1 Performance Benchmarking Tooling As a performance benchmark should closely reflect production traffic, everyone’s benchmarking setup should look a bit different. But there are a few common tools: - **SGLang Genai-bench:** A CLI and dashboard by the SGLang team for benchmarking models deployed with any inference framework. - **NVIDIA GenAI-Perf:** A client-side tool by NVIDIA for measuring latency and throughput on varied traffic. - **Locust:** An open-source load-testing tool, not specific to generative AI systems, that simulates as many as millions of simultaneous users. Another great tool for benchmarking is open-source evals datasets – from general evals like MMLU and gsm8k to domain-specific evals like SWE-bench. While the purpose of benchmarking work is to measure performance, not model output quality, these eval datasets serve two purposes: acting as a set of varied and realistic inputs, and spot checking that performance optimizations haven’t impacted model output quality. When possible, choose an eval dataset that matches the expected use of your production system, like HumanEval when reducing latency for a code completion system. ### 4.5.2 Performance Benchmarking Tips Along with being realistic, great benchmarks are also consistent. Make sure your benchmarks send enough traffic to get a good read on performance without being swayed by outliers. When in doubt, run a benchmark multiple times and average the results. Before you do any performance optimization work, start with a solid baseline benchmark. As you test optimizations, keep a consistent configuration in your benchmarking setup, and test each optimization individually as well as collectively to fully understand what is driving performance improvement. In some cases, optimizations can work against each other, like trying to run speculative decoding with large batch sizes. The principle of changing one thing at a time applies to your benchmarking configuration as well. It’s common to need to test various traffic patterns or sequence shapes, but as with any experiment only change one variable at a time to ensure that you are getting clear results. ### 4.5.3 Profiling Performance Profiling is one click deeper than benchmarking. Where a benchmark gives a single figure (e.g., the P90 TTFT is 350 ms), a profiling tool shows where each of those milliseconds was spent in the inference process. Benchmarking tells you how your system is performing; profiling tells you why it’s performing that way. ![Figure 4.7: A kernel profiler shows you how long each operation within a kernel takes to execute, revealing bottlenecks.](https://www.datocms-assets.com/104802/1788123368-inference-engineering-figure-4-7.png) _Figure 4.7: A kernel profiler shows you how long each operation within a kernel takes to execute, revealing bottlenecks._ Most inference engineers won’t need to do profiling as part of their daily work. When using an already high-performance tool like the inference engine TensorRT-LLM, your workflow is a cycle of configuration and benchmarking – profiling would be extraneous. However, if you’re contributing to an inference framework like vLLM or SGLang, writing your own inference service in PyTorch, or operating at the cutting edge of a new modality like video generation, performance profiling should be part of your toolkit. The most popular profiling tools for inference are: - **PyTorch Profiler**: An easy-to-use profiling library for capturing step-by-step performance metrics (CPU time, GPU time, memory usage) during inference. - **NVIDIA Nsight Systems (NSys)**: A featureful but complex tool for GPU and CPU sampling and tracing that provides system-wide analysis across multiple GPUs and their interconnects. - **NVIDIA Nsight Compute (NCU)**: A profiling utility and CLI for in-depth analysis of individual CUDA kernels on both compute and memory usage. In addition, frameworks like TensorFlow and TensorRT ship with their own built-in profilers. Profilers are valuable because they give you granular information about compute and memory usage, which guides your optimization work toward improving the most expensive steps in your inference pipeline. For example, using PyTorch Profiler you might find that activation functions are taking an unusually long time due to excess memory reads, and figure out how to write a fused kernel that runs activations alongside attention to prevent the excess reads. Then, you would insert that new kernel into your PyTorch code and re-run system-level benchmarks to see if you’ve achieved your latency targets. Together, profiling and benchmarking give you the information you need to improve system performance and, eventually, the confidence to deploy your optimizations in production.