NVIDIA Dynamo is a distributed system for model serving first announced at NVIDIA GTC in March 2025.
Dynamo works with every inference engine – vLLM, SGLang, and TensorRT-LLM – as backends, with Dynamo itself providing an orchestration layer for large-scale deployments.
Dynamo provides support for essential model performance techniques:
- KV cache re-use: Retaining KV information between requests and routing requests based on prefix match.
- Disaggregation: Separating prefill and decode onto individually optimized engines with independent scaling.
- Multi-node parallelism: Optionally using two or more nodes of GPUs in a single replica for a model, usually with Expert Parallelism.
Each of these techniques will be detailed in chapter 5. As with the inference engines, there is a lot of work for inference engineers to do to configure Dynamo for their use case and achieve maximum performance.
Dynamo’s thoughtful abstractions for distributed KV routing, disaggregation, and multi-node model parallelism provide high-performance aggregation of information during runtime, allowing for real-time adjustments to configuration as traffic fluctuates. For example, you can automatically scale up and down prefill and decode workers with an SLA-based planner operating on user-defined TTFT and TPS constraints.
As a general principle, the more scale you have, the more tools and techniques there are available to you for inference optimization.
Dynamo is built for scale: big models, big traffic. It excels at serving foundation models like the trillion-parameter Kimi family to a large number of concurrent users. For smaller models, Dynamo can still offer moderate performance improvements on large-scale deployments.
If you’re building an inference API for a built-from scratch foundation model or serving an open model in a high-usage product, Dynamo is a great choice.
But many deployments don’t need the additional complexity of Dynamo. Unless you’re operating with enough volume for disaggregation and KV-aware routing to matter, Dynamo will be unnecessary work and excess overhead. In these cases, you can use inference engines directly.
Dynamo is the newest project covered in this chapter, and features are still being built out. Dynamo is open source under the Apache 2.0 license. The community around Dynamo is active, and the project welcomes contributions with a public CI and support from NVIDIA engineers.