Benchmarking is an essential part of model performance optimization. Without precise, accurate performance benchmarks, there’s no way of knowing if your optimizations are actually working.
A high-quality benchmark simulates real life as closely as possible. The best benchmark is to shadow real-world production traffic onto the system you are testing. Shadowing is the process of copying incoming requests onto the test system so that you can benchmark its performance without affecting the original request.
If you can’t shadow real usage, you’ll need to simulate it. LLM performance is affected by a number of factors. When simulating traffic, you need to match your expected production workload on multiple dimensions:
- Sequence lengths: Time to first token and memory usage rely on the input sequence length (ISL) and output sequence length (OSL), meaning the number of tokens in the prompt and response.
- Volume and pattern of traffic: Batching and server load depend on the number of concurrent requests. Jitter traffic to mimic real usage.
- Request contents: The actual prompt within each request affects performance factors like cache hit rate and draft token acceptance.
- Input parameters: Settings like temperature and reasoning effort that affect inference should be set to their anticipated production values.
Remember, optimization is about tradeoffs and constraints. If you’re maximizing benchmark performance against bad inputs, performance in production won’t match expectations.
4.5.1 Performance Benchmarking Tooling
As a performance benchmark should closely reflect production traffic, everyone’s benchmarking setup should look a bit different. But there are a few common tools:
- SGLang Genai-bench: A CLI and dashboard by the SGLang team for benchmarking models deployed with any inference framework.
- NVIDIA GenAI-Perf: A client-side tool by NVIDIA for measuring latency and throughput on varied traffic.
- Locust: An open-source load-testing tool, not specific to generative AI systems, that simulates as many as millions of simultaneous users.
Another great tool for benchmarking is open-source evals datasets – from general evals like MMLU and gsm8k to domain-specific evals like SWE-bench.
While the purpose of benchmarking work is to measure performance, not model output quality, these eval datasets serve two purposes: acting as a set of varied and realistic inputs, and spot checking that performance optimizations haven’t impacted model output quality.
When possible, choose an eval dataset that matches the expected use of your production system, like HumanEval when reducing latency for a code completion system.
4.5.2 Performance Benchmarking Tips
Along with being realistic, great benchmarks are also consistent. Make sure your benchmarks send enough traffic to get a good read on performance without being swayed by outliers. When in doubt, run a benchmark multiple times and average the results.
Before you do any performance optimization work, start with a solid baseline benchmark. As you test optimizations, keep a consistent configuration in your benchmarking setup, and test each optimization individually as well as collectively to fully understand what is driving performance improvement. In some cases, optimizations can work against each other, like trying to run speculative decoding with large batch sizes.
The principle of changing one thing at a time applies to your benchmarking configuration as well. It’s common to need to test various traffic patterns or sequence shapes, but as with any experiment only change one variable at a time to ensure that you are getting clear results.
4.5.3 Profiling Performance
Profiling is one click deeper than benchmarking. Where a benchmark gives a single figure (e.g., the P90 TTFT is 350 ms), a profiling tool shows where each of those milliseconds was spent in the inference process. Benchmarking tells you how your system is performing; profiling tells you why it’s performing that way.
Most inference engineers won’t need to do profiling as part of their daily work. When using an already high-performance tool like the inference engine TensorRT-LLM, your workflow is a cycle of configuration and benchmarking – profiling would be extraneous.
However, if you’re contributing to an inference framework like vLLM or SGLang, writing your own inference service in PyTorch, or operating at the cutting edge of a new modality like video generation, performance profiling should be part of your toolkit.
The most popular profiling tools for inference are:
- PyTorch Profiler: An easy-to-use profiling library for capturing step-by-step performance metrics (CPU time, GPU time, memory usage) during inference.
- NVIDIA Nsight Systems (NSys): A featureful but complex tool for GPU and CPU sampling and tracing that provides system-wide analysis across multiple GPUs and their interconnects.
- NVIDIA Nsight Compute (NCU): A profiling utility and CLI for in-depth analysis of individual CUDA kernels on both compute and memory usage.
In addition, frameworks like TensorFlow and TensorRT ship with their own built-in profilers.
Profilers are valuable because they give you granular information about compute and memory usage, which guides your optimization work toward improving the most expensive steps in your inference pipeline.
For example, using PyTorch Profiler you might find that activation functions are taking an unusually long time due to excess memory reads, and figure out how to write a fused kernel that runs activations alongside attention to prevent the excess reads. Then, you would insert that new kernel into your PyTorch code and re-run system-level benchmarks to see if you’ve achieved your latency targets.
Together, profiling and benchmarking give you the information you need to improve system performance and, eventually, the confidence to deploy your optimizations in production.