# 4.3 Inference Engines _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 4: Software. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) There are three competitive inference engines on the market: vLLM, SGLang, and TensorRT-LLM. These frameworks offer good out-of-the-box performance for LLMs and other modalities with similar architectures (chapter 6). In late 2025, vLLM and SGLang also began supporting some image and video generation models via vLLM Omni and SGLang Diffusion, respectively. TensorRT-LLM does not support image or video generation models. Inference engineers can also use TensorRT or PyTorch directly to run these models (section 6.5). Inference engines are powerful because they are configurable. Working with pre-optimized components at a higher level of abstraction, inference engineers can spend their time testing combinations of techniques rather than repeating routine implementations. At a very high level, vLLM and SGLang are more general tools that are easier to adopt and have day zero support for more models, while TensorRT-LLM has a steeper learning curve but usually achieves the best performance. | Engine | vLLM | SGLang | TensorRT-LLM | | :------------ | :--------- | :---------- | :----------- | | Performance | Good | Good | Best | | Ease of use | Easy | Easy | Hard | | Model support | Most | Most | Some | | Hardware | GPU, TPU | NVIDIA, AMD | NVIDIA only | | License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Each framework runs out of the box with core features like continuous batching and supports the main performance optimization techniques – post-training quantization, speculative decoding, prefix caching, parallelism, disaggregation. At Baseten, we use all three frameworks, though we use TensorRT-LLM the most. Inference engineers should be familiar with all three and select on a deployment-by-deployment basis. ## 4.3.1 vLLM vLLM has the largest market share among inference engines. GitHub stars are a rough measure for popularity, but at the time of publication vLLM has twice as many stars as SGLang and TensorRT-LLM combined. First released in the summer of 2023, vLLM is the oldest of these inference engines by a few months. Originally created at UC Berkeley, vLLM is now hosted by The PyTorch Project within The Linux Foundation. vLLM’s best selling point is its broad support. It supports the most hardware options – NVIDIA, AMD, and Intel GPUs along with Google TPUs – as well as the most models and architectures. Just about every open LLM out there integrates with vLLM from day zero. vLLM also supports multimodal inference via vLLM Omni, which extends the engine to support image, audio, and video inputs and outputs. One of the core principles of inference engineering is that the more constraints you can introduce, the better performance you can achieve. vLLM’s broad platform can achieve impressive performance results when properly configured, but in my experience it falls short of the highest-end performance possible with narrow frameworks like TensorRT-LLM. vLLM’s developer experience is built around the `vllm serve` command, with server configuration passed in as flags. ![Figure 4.4: vLLM inference example on eight GPUs.](https://www.datocms-assets.com/104802/1788123353-inference-engineering-figure-4-4.png) _Figure 4.4: vLLM inference example on eight GPUs._ vLLM is pip-installable and provides official Docker images with pre-bundled dependencies and support for various hardware architectures. You should use vLLM when: - You want to quickly stand up a model server that will offer solid performance out of the box for almost any open model. - You want to run an “Omni” model with multiple input and output modalities. - You are using a smaller GPU or older architecture where TensorRT-LLM offers few performance benefits. ## 4.3.2 SGLang SGLang is the other major community-driven fast inference framework. First released in December 2023, SGLang has risen to prominence alongside Chinese open models like DeepSeek and Qwen and is the engine of choice for inference at xAI. SGLang’s unique angle on the problem of model serving is expressed in its developer experience, which pairs a fast backend runtime with a flexible frontend language. In practice, that means you can choose individual components of your engine for deep customization without needing to rewrite everything else from scratch. SGLang supports both NVIDIA and AMD GPUs, and has strong day-zero support for a wide range of models. SGLang works closely with labs like DeepSeek, Qwen, Kimi, and Z AI to release optimized implementations of new architectural features like DeepSeek’s Multi-Latent Attention. SGLang has invested heavily in supporting large-scale deployments of MoE LLMs, specifically multi-node deployments on systems like GB200 NVL72 for high throughput. These systems offer extremely cost-efficient inference for large models with significant traffic. SGLang’s developer experience is built around the `sglang.launch_server` command, with server configuration passed in as flags. ![Figure 4.5: SGLang inference example on eight GPUs.](https://www.datocms-assets.com/104802/1788123358-inference-engineering-figure-4-5.png) _Figure 4.5: SGLang inference example on eight GPUs._ SGLang also supports image and video generation model inference via SGLang Diffusion. SGLang Diffusion introduces a pipeline abstraction which orchestrates a number of stages. This flexible approach maps closely to the architecture of image and video generation models. For performance, SGLang Diffusion adds support for various diffusion-specific parallelism methods and re-uses the scheduler and optimized kernels from the main SGLang package. You should use SGLang when: - You want excellent out-of-the-box throughput with decent latency on large MoE models like DeepSeek and Kimi. - You want the inference engine experience for image and video generation models. - You want control and customization and are excited to participate in the SGLang community. ## 4.3.3 TensorRT-LLM TensorRT-LLM is NVIDIA’s open-source inference engine. Of the three main options, TensorRT-LLM offers the highest performance and the most flexibility to expert users. A note on naming: There are two major versions of TensorRT-LLM. Only the older version is actually related to TensorRT: - **TensorRT-LLM V0 (0.X.Y):** Major versions starting with zero are a plugin for NVIDIA TensorRT. - **TensorRT-LLM V1 (1.X.Y):** Major versions starting with one are a standalone package based on PyTorch with no dependency on TensorRT. Originally, TensorRT-LLM built a TensorRT engine for serving language models. With the modern PyTorch-based version, TensorRT-LLM bypasses the intermediate representation of TensorRT and uses PyTorch directly. TensorRT-LLM V1 was released in the summer of 2025. Deployments of the previous major version are still common – always be sure to check which version you are using. TensorRT-LLM achieves the best performance in large part because it has access to kernels written by NVIDIA engineers, including some closed-source kernels. These handwritten and manually fused kernels offer excellent support for the latest hardware architectures like Hopper and Blackwell and for NVIDIA-specific number formats like NVFP4. TensorRT-LLM offers a robust implementation of in-flight batching (token-level continuous batching), which helps with throughput. It also supports just about every model performance optimization setting you could ask for, including quantization, speculation algorithms, prefix caching, chunked prefill, flexible parallelism, and disaggregation. With V1, TensorRT-LLM introduces a developer experience that looks a lot like vLLM and SGLang. However, in addition to flag arguments on the `trtllm-serve` command, it expects a `config.yaml` file for deeper customization. ![Figure 4.6: TensorRT-LLM inference example on eight GPUs.](https://www.datocms-assets.com/104802/1788123363-inference-engineering-figure-4-6.png) _Figure 4.6: TensorRT-LLM inference example on eight GPUs._ The best way to install TensorRT-LLM is by running it via one of NVIDIA’s official Docker containers Use TensorRT-LLM when: - You are running a well-supported model architecture on a Hopper or later GPU. - You are willing to do extra engineering work to get the best possible performance. - Optionally, you are planning to use NVIDIA Dynamo for serving and want the most deeply integrated engine.