Chapter 7: Production

7.1 Containerization

Packaging model servers so deployments are reproducible and start quickly.

Containerization is the practice of packaging an application together with its dependencies to standardize deployment in production. Containers turn a program into a packaged artifact that can run anywhere – no more “it works on my machine.”

Containers are lightweight because they share the underlying host operating system kernel (in this context, a kernel refers to a Linux kernel, not a CUDA kernel). This makes containers well-suited for packaging inference services.

For most developers, containerization is synonymous with Docker. Working with containers introduces more specific terminology:

  • Container: An actively running environment that isolates an application and its dependencies.
  • Image: An executable package that contains everything you need to run a piece of software.
  • Dockerfile: A human-readable file with well-specified, machine-interpretable instructions for creating an image.
  • Registry: A central repository for managing, storing, sharing, and distributing images.

NVIDIA, several cloud providers, and Docker themselves all operate container registries. A popular registry for AI is Docker Hub – Docker Hub is to images as Hugging Face is to model weights or PyPi is to Python packages.

Docker containers are composed of layers. You can take a pre-configured base image and add other layers with additional filesystem changes on top of it.

Figure 7.1: Docker containers are composed of layers, from a base image up to an ephemeral, writable top layer.
Figure 7.1: Docker containers are composed of layers, from a base image up to an ephemeral, writable top layer.

There are three types of layers:

  • Base image: Either an operating system distribution like Ubuntu or a more complex image taken from a registry. The base image itself is composed of multiple layers.
  • Additional layers: Filesystem changes including dependencies, application code, and configuration files as specified by Dockerfile instructions.
  • Container layer: A thin, ephemeral layer created at runtime. Any changes to the running container, like creating, updating, or deleting files, are written to this layer and are lost when the container terminates.

Inference engines like vLLM and SGLang offer official base images for active releases. It’s generally a good idea to start from one of these proven images, rather than building your own from scratch.

7.1.1 Dependency Management

Dependency chains for inference are long and fragile. Getting to a working build is hard, which makes containerization essential for preserving a known good build in an ecosystem where breaking changes are all too common.

Images are built for a specific GPU architecture and model. A container includes many runtime components:

  • CUDA toolkit version: The specific versions of CUDA, cuDNN, and drivers compatible with the rest of your stack.
  • Python packages: Dependencies like torch, transformers, and diffusers.
  • Inference engine: The version of vLLM, SGLang, TensorRT-LLM, or any other inference engine used.
  • System packages: Linux packages like ffmpeg, especially common when working with audio, image, or video models.

Like a hiker on a backpacking trip, you want to pack light. Images built for inference are often many gigabytes. For fast deployment and efficient operation, only include strictly necessary dependencies.

Another best practice is pinning versions. Having a pinned dependency tree keeps the system runtime behavior consistent across different environments and enables repeated builds of the image with the same result. Specify exactly which version of each dependency should be included in the image.

Figure 7.2: Requirements should be pinned to exact versions to prevent future changes from breaking inference containers.
Figure 7.2: Requirements should be pinned to exact versions to prevent future changes from breaking inference containers.

Tools like uv, poetry, or pip will flag any version incompatibilities and throw an error when building an image. With pinned versions, once an image is successfully built once, it will always resolve dependencies to the same versions and protect you against breaking changes.

Breaking changes are particularly common when working with newly released models. When new versions of models like DeepSeek are announced, the entire inference ecosystem races to offer day zero support.

When building images for brand-new models, inference engineers often rely on overnight builds or other developer pre-releases for dependencies rather than stable releases. These early versions are more prone to bugs and often need to be rebuilt on stable releases in the days and weeks following the model drop.

7.1.2 NIMs

NVIDIA Inference Microservices (NIMs) are pre-built Docker containers for popular open models.

Containers make inference service implementations portable. NVIDIA created two types of NIMs:

  • Multi-LLM NIM: A flexible container for running a family of models on a supported GPU architecture.
  • LLM-specific NIM: An engine optimized for a specific model on a specific GPU configured for maximum performance.

NIMs are available for various common combinations of model, GPU architecture, GPU count, and configuration.

A NIM is like any other container. You can use a NIM as a starting point to build on, as a reference architecture to learn from, or as an out-of-the-box inference service.

However, if you’re looking for maximum control instead of a done-for-you configuration, you’ll generally be better off building your own container from a less opinionated base image rather than adapting a NIM.