# Chapter 7: Production _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) The purpose of inference engineering is to make generative AI models faster, less expensive, and more reliable to operate, which in turn lets you build better products. This promise is only fulfilled if your inference engineering work makes it to production and scales alongside the hypergrowth and viral spikes that successful AI products generate. When you scale production traffic, your assumptions are rigorously tested. Everything from sequence shapes to traffic patterns to what topic a user decides to chat about impacts your observed performance in production. And maintaining secure, robust infrastructure is an entirely different skillset from optimizing model inference on the GPU. No matter how fast and efficiently a single instance can serve a model, with enough traffic, the service will be overwhelmed. That is not a PyTorch problem or a CUDA problem, it’s an infrastructure problem, and requires a different mindset and different technologies. Scaling in production introduces new complexities about where and how to get GPUs, balance traffic across them, and prevent downtime. Plus, cost accounting gets messy in the transition from paying per million tokens to paying directly for infrastructure. Latency in production comes from more than just prefill and decode. You need to evaluate your system end-to-end and eliminate inefficiencies in the server, the network, and even the client in situations where you can own or influence client code. This chapter introduces the essential considerations for scaling low-latency, high-throughput inference in production. And at the end, I’ll invite you to try Baseten for deploying mission-critical inference workloads. ## 7.1 Containerization Containerization is the practice of packaging an application together with its dependencies to standardize deployment in production. Containers turn a program into a packaged artifact that can run anywhere – no more “it works on my machine.” Containers are lightweight because they share the underlying host operating system kernel (in this context, a kernel refers to a Linux kernel, not a CUDA kernel). This makes containers well-suited for packaging inference services. For most developers, containerization is synonymous with Docker. Working with containers introduces more specific terminology: - **Container:** An actively running environment that isolates an application and its dependencies. - **Image:** An executable package that contains everything you need to run a piece of software. - **Dockerfile:** A human-readable file with well-specified, machine-interpretable instructions for creating an image. - **Registry:** A central repository for managing, storing, sharing, and distributing images. NVIDIA, several cloud providers, and Docker themselves all operate container registries. A popular registry for AI is Docker Hub – Docker Hub is to images as Hugging Face is to model weights or PyPi is to Python packages. Docker containers are composed of layers. You can take a pre-configured base image and add other layers with additional filesystem changes on top of it. ![Figure 7.1: Docker containers are composed of layers, from a base image up to an ephemeral, writable top layer.](https://www.datocms-assets.com/104802/1788123527-inference-engineering-figure-7-1.png) _Figure 7.1: Docker containers are composed of layers, from a base image up to an ephemeral, writable top layer._ There are three types of layers: - **Base image:** Either an operating system distribution like Ubuntu or a more complex image taken from a registry. The base image itself is composed of multiple layers. - **Additional layers:** Filesystem changes including dependencies, application code, and configuration files as specified by Dockerfile instructions. - **Container layer:** A thin, ephemeral layer created at runtime. Any changes to the running container, like creating, updating, or deleting files, are written to this layer and are lost when the container terminates. Inference engines like vLLM and SGLang offer official base images for active releases. It’s generally a good idea to start from one of these proven images, rather than building your own from scratch. ### 7.1.1 Dependency Management Dependency chains for inference are long and fragile. Getting to a working build is hard, which makes containerization essential for preserving a known good build in an ecosystem where breaking changes are all too common. Images are built for a specific GPU architecture and model. A container includes many runtime components: - **CUDA toolkit version:** The specific versions of CUDA, cuDNN, and drivers compatible with the rest of your stack. - **Python packages:** Dependencies like `torch`, `transformers`, and `diffusers`. - **Inference engine:** The version of vLLM, SGLang, TensorRT-LLM, or any other inference engine used. - **System packages:** Linux packages like `ffmpeg`, especially common when working with audio, image, or video models. Like a hiker on a backpacking trip, you want to pack light. Images built for inference are often many gigabytes. For fast deployment and efficient operation, only include strictly necessary dependencies. Another best practice is pinning versions. Having a pinned dependency tree keeps the system runtime behavior consistent across different environments and enables repeated builds of the image with the same result. Specify exactly which version of each dependency should be included in the image. ![Figure 7.2: Requirements should be pinned to exact versions to prevent future changes from breaking inference containers.](https://www.datocms-assets.com/104802/1788123536-inference-engineering-figure-7-2.png) _Figure 7.2: Requirements should be pinned to exact versions to prevent future changes from breaking inference containers._ Tools like uv, poetry, or pip will flag any version incompatibilities and throw an error when building an image. With pinned versions, once an image is successfully built once, it will always resolve dependencies to the same versions and protect you against breaking changes. Breaking changes are particularly common when working with newly released models. When new versions of models like DeepSeek are announced, the entire inference ecosystem races to offer day zero support. When building images for brand-new models, inference engineers often rely on overnight builds or other developer pre-releases for dependencies rather than stable releases. These early versions are more prone to bugs and often need to be rebuilt on stable releases in the days and weeks following the model drop. ### 7.1.2 NIMs NVIDIA Inference Microservices (NIMs) are pre-built Docker containers for popular open models. Containers make inference service implementations portable. NVIDIA created two types of NIMs: - **Multi-LLM NIM:** A flexible container for running a family of models on a supported GPU architecture. - **LLM-specific NIM:** An engine optimized for a specific model on a specific GPU configured for maximum performance. NIMs are available for various common combinations of model, GPU architecture, GPU count, and configuration. A NIM is like any other container. You can use a NIM as a starting point to build on, as a reference architecture to learn from, or as an out-of-the-box inference service. However, if you’re looking for maximum control instead of a done-for-you configuration, you’ll generally be better off building your own container from a less opinionated base image rather than adapting a NIM. ## 7.2 Autoscaling The goal of autoscaling is to ensure that you always have enough resources to serve all incoming requests while maintaining your latency SLAs without wasting money on idle GPUs. ![Figure 7.3: Without autoscaling, inference systems waste resources during traffic lulls and miss SLAs during traffic spikes.](https://www.datocms-assets.com/104802/1788123541-inference-engineering-figure-7-3.png) _Figure 7.3: Without autoscaling, inference systems waste resources during traffic lulls and miss SLAs during traffic spikes._ ![Figure 7.4: A strong autoscaling system for inference matches resources to demand.](https://www.datocms-assets.com/104802/1788123547-inference-engineering-figure-7-4.png) _Figure 7.4: A strong autoscaling system for inference matches resources to demand._ Autoscaling systems use Kubernetes, an open-source container orchestration system, along with a cluster-level system for provisioning and deallocating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires. Kubernetes works by composing a group of hardware resources together into a cluster. This cluster has two types of components: - **Control plane:** Makes routing and scaling decisions. - **Worker plane:** Runs the actual containerized applications. ![Figure 7.5: Kubernetes clusters have a single control plane that orchestrates multiple workers.](https://www.datocms-assets.com/104802/1788123555-inference-engineering-figure-7-5.png) _Figure 7.5: Kubernetes clusters have a single control plane that orchestrates multiple workers._ A Kubernetes cluster can run multiple replicas of multiple models. But how do you decide how many replicas of each model to run? Unless your traffic is unusually consistent, there probably isn’t one number of replicas that perfectly matches your needs. Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions: - **Utilization:** Scale up and down based on GPU utilization signals like memory usage or compute usage. - **Traffic:** Scale up and down based on the number of requests being processed in the system. Utilization and traffic don’t always match. For example, in LLM prefill, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates. Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand. When designing a traffic-based autoscaling system, you want to configure five factors: - **Min replicas:** What is the minimum number of replicas that stay running, regardless of traffic? - **Max replicas:** What is the maximum number of replicas that you can allocate when traffic is high? - **Autoscaling window:** How long is the sliding timeframe that you use to measure traffic and make autoscaling decisions? - **Scale down delay:** How long after a scale down is suggested do you wait in case of another traffic spike? - **Concurrency target:** How many requests can each replica handle at once? The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale down delay prevents premature scaledowns for spikey traffic, but could result in unnecessary spend after traffic truly cools off. ### 7.2.1 Concurrency and Batch Sizing To properly operate a traffic-based autoscaling system, you need a strong understanding of how much concurrent traffic each instance can handle. Most model inference services can handle more than one request at a time via batching. There are several approaches to batching: - **Static batching:** The server waits until the batch is full before starting inference. - **Dynamic batching:** The server waits until the batch is full or a configured amount of time has passed before starting inference. - **Continuous batching:** The server continuously runs inference, swapping in requests as slots become available. Inference engines like vLLM, SGLang, and TensorRT-LLM implement robust continuous batching (or in-flight batching as TensorRT-LLM calls it), where requests are batched at the token level. This minimizes latency relative to static batching. ![Figure 7.6: Static batching sets a fixed batch size and waits for the batch to fill before beginning inference, leading to long wait times for early requests.](https://www.datocms-assets.com/104802/1788123563-inference-engineering-figure-7-6.png) _Figure 7.6: Static batching sets a fixed batch size and waits for the batch to fill before beginning inference, leading to long wait times for early requests._ ![Figure 7.7: Dynamic batching adds a cutoff time after which a batch is run whether or not it is full.](https://www.datocms-assets.com/104802/1788123572-inference-engineering-figure-7-7.png) _Figure 7.7: Dynamic batching adds a cutoff time after which a batch is run whether or not it is full._ ![Figure 7.8: Continuous batching operates at the token level, switching in new requests as old requests finish.](https://www.datocms-assets.com/104802/1788123577-inference-engineering-figure-7-8.png) _Figure 7.8: Continuous batching operates at the token level, switching in new requests as old requests finish._ Batch sizing trades off latency for throughput. Increasing the batch size will produce more throughput overall, but each user’s latency will get worse. Test performance across multiple batch sizes to find the right fit for your model, instance, latency target, and budget. This is controlled at the autoscaling configuration level via the concurrency target and at the replica level via the batch size, which should match. Once every active replica reaches its maximum concurrency, the autoscaling system knows to spin up more replicas. If enough replicas are kicking off half-full batches, it’s time to scale back down. ### 7.2.2 Cold Starts A cold start is the time it takes to spin up a new replica of a model. The overall performance of an autoscaling system depends on its cold start speeds. If you can’t spin up replicas fast, it’s hard to confidently scale down, leading to over-provisioning. There are several factors that affect cold start times: - **GPU procurement:** How quickly can you add the necessary GPUs to your cluster and allocate them to the model? - **Image loading:** How quickly can you load the container image onto the newly procured instance? - **Model loading:** How quickly can you load the model weights into the container? - **Engine startup:** How quickly can you start your inference engine, including any compilation time? Each of these factors needs to be optimized separately. ![Figure 7.9: Each step in the cold start process adds to the overall timeline.](https://www.datocms-assets.com/104802/1788123582-inference-engineering-figure-7-9.png) _Figure 7.9: Each step in the cold start process adds to the overall timeline._ Unless you have a pool of warm nodes that you’re flexing between models, GPU procurement speed is mostly a function of your cloud provider. Section 7.3.1 covers procuring GPUs, and node start time is one of the negotiable factors in a contract. However, engineers can do a lot on loading images and weights and starting containers and engines. Loading images and model weights is a function of how quickly you can write gigabytes, often hundreds of gigabytes, of data onto your instance. There are two ways to load images and weights faster: make them smaller or get more bandwidth. Including only necessary steps and dependencies makes images smaller and faster to build into containers, while quantizing model weights has the additional benefit of making them faster to load during cold starts. For small models, the strategy used to be baking the weights into the image to simplify caching and loading. However, now that most models have dozens or hundreds of billions of parameters, the weights dwarf the image and are better loaded separately. Where you load weights from has a massive impact on the bandwidth. If you’re loading from a third party like Hugging Face, you’re limited by their egress speed. And storing your weights in an S3 bucket introduces network latency and data transfer costs. For loading multi-hundred-billion-parameter models, you need gigabytes per second of bandwidth. The best way to get this is by loading over network within a node from a source cached physically near the GPU instance within the same datacenter. Inference engines like vLLM and SGLang are fast to start up. But engines like TensorRT-LLM and optimized models with PyTorch have a compilation step that targets the specific hardware resources to build the model inference engine. These compilations often take several minutes. In these cases, caching built engines massively improves cold start times. Both TensorRT-LLM and PyTorch have image caching mechanisms that make this possible, though you’ll always need to load a cached engine into an instance with exactly the GPU type, CUDA version, and software dependencies as the environment that the engine was built in for it to run properly. ### 7.2.3 Routing, Load Balancing, and Queueing Once there are multiple replicas online, the system needs to make a decision about which requests to send to which replicas. There are two types of components that make these decisions: - **Routers:** A router works at the request level to determine the ideal place to send a given request. A router answers “where should this request go?” - **Load balancers:** A load balancer works at the system level to even out requests between multiple options. A load balancer answers “where could this request go?” In complex systems, there isn’t just one router and one load balancer. Routing occurs throughout the stack with load balancers injected at key points to keep system-wide performance stable. Overall, you want to split load equally across replicas. However, routing and load balancing are not as simple as saying, “Well, I have 3 replicas and 12 requests, so let’s put 4 requests on each replica.” Each request may have a different number of input tokens. If most requests have 100 input tokens, a request with 10,000 will unbalance a simple system. Some requests are also better handled by certain replicas. Examples include: - **KV cache-aware routing:** Direct a request to a replica that already has a matching prefix in its KV cache. - **LoRA-aware routing:** Direct a request to a replica that already has the desired LoRA fine-tuned weights in memory. Intelligent request routing uses information from the inference engine and any orchestrators like NVIDIA Dynamo to route requests based on sequence length, prefix, and LoRA needs. Load balancing and routing are not enough. When an autoscaling system receives more traffic than it can handle, it needs a way to hold onto requests as it scales up more resources or waits for existing resources to become available. A queue is the infrastructure primitive for handling this situation. A standard queue is a first-in, first-out system for excess requests, though you can do a more complex implementation like a priority queue to, for example, give paid users priority over free users in high-traffic scenarios. As new replicas come online, ensure that the queue sees them and requests don’t continue waiting for the existing replicas. Each new replica should immediately be assigned up to its concurrency limit in queued traffic once active. ### 7.2.4 Scale to Zero Advanced autoscaling systems implement a mechanism for scale to zero, where the system can scale down to zero active replicas if there is no traffic, then scale up automatically when traffic is received. Scale to zero relies on two prerequisites: - **Fast cold starts:** As users may be waiting live, cold starts must be as fast as possible. - **Robust queueing:** The system needs to be able to hold the incoming requests until a replica is live. Even with these capabilities, scale to zero is not a fit for all workloads. Scale to zero is great for development, when testing is bursty and latency for the first request is unimportant. And in production, scale to zero is useful for applications that only get traffic periodically, like an agent that’s only accessed during business hours in one country or an offline system designed for daily batch processing jobs. However, if you’re relying on scale to zero to keep costs low in a latency-sensitive application that gets light, unscheduled traffic, it’s probably a sign that your AI application is not yet ready for dedicated infrastructure and should use pay-per-token APIs until greater scale is reached. ### 7.2.5 Independent Component Scaling AI applications are increasingly built on multi-model, multi-stage compound AI workloads where inference engineers need to coordinate multiple steps to fulfill a single request. These steps may have different hardware needs. A voice activity detector model needs a far less powerful GPU than the transcription model it’s chunking data for, while the LLM processing that transcript may need a full node with multiple GPUs. And the scaling parameters for each step in the pipeline also differ. ![Figure 7.10: Independent component scaling gives each model access to appropriate resources and individual scaling.](https://www.datocms-assets.com/104802/1788123587-inference-engineering-figure-7-10.png) _Figure 7.10: Independent component scaling gives each model access to appropriate resources and individual scaling._ For these pipelines, you need to decompose autoscaling decisions and scale each step individually to right-size resources per step and avoid both bottlenecks and overprovisioning. However, every model in the pipeline should run in the same cluster. If it takes 10 milliseconds to send a message within a cluster and 50 milliseconds to send messages between clusters, that 40-millisecond difference across a 5-step pipeline would be 20 percent of a one-second latency SLA. ## 7.3 Multi-Cloud Capacity Management Autoscaling within a single cluster works up to a certain point. But high-volume deployments serving a global user base need thousands of GPUs distributed around the world. It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fluidly, and moving workloads across clouds is a tedious, error-prone process. True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling self-healing and global scheduling. ![Figure 7.11: A multi-cloud approach extends the idea of control and workload planes to a multi-cluster, multi-region system.](https://www.datocms-assets.com/104802/1788123081-inference-engineering-figure-0-3.png) _Figure 7.11: A multi-cloud approach extends the idea of control and workload planes to a multi-cluster, multi-region system._ Running true multi-cloud inference unlocks: - **Capacity:** Pool capacity from multiple providers for greater and more flexible GPU access. - **Redundancy:** Split inference across providers for resiliency against outages. - **Latency:** Run inference close to your end users to reduce network latency overhead. - **Compliance:** Run inference in compliance with data sovereignty and other regulatory requirements. Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains: - **Control plane:** Handles model deployment and global scaling decisions, receives real-time event streams. - **Workload planes:** Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand. This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected. ### 7.3.1 GPU Procurement There are a good number of companies in the business of providing access to GPUs. The three major types are: - **Hyperscalers:** Large cloud providers like AWS or GCP. - **Neoclouds:** GPU-focused clouds like Coreweave or Nebius. - **Resellers:** Secondary markets like SF Compute Company. Players in this space vary in their capacity, availability, and reliability. You generally pay a premium for hyperscalers and across all providers there is a tradeoff between cost and factors like uptime SLAs, support, regional availability, instance configuration, and cluster sizes. The first challenge is securing capacity. It is often difficult to get your hands on the GPUs you need, especially the latest hardware. Large clusters are also hard to get, with relatively few players offering blocks of hundreds of nodes. Many cloud providers allocate the majority of in-demand GPUs to their largest customers on long-term reservations. You may need to work across multiple cloud providers to get the GPUs you need in the right regions. Cloud GPUs can be procured via three different mechanisms: - **Reserved:** Blocks of hundreds or thousands of GPUs are reserved for months or years at discounted rates. - **On-demand:** Individual instances are available as needed up to a given quota for a relatively high per-hour cost. - **Spot:** Discounted on-demand instances that can be pre-empted at an agreed-upon notice period, often minutes. Large-scale inference generally uses a blend of GPU sources, with a baseline of low-cost reserved instances and a mix of on-demand and spot for handling peaks in traffic. These GPUs are distributed across multiple clusters worldwide for proximity to end users. ### 7.3.2 Geo-Aware Load Balancing Successful AI applications have users all over the world. Just like an individual cluster has a load balancer to ensure that every GPU in the cluster receives the right amount of traffic, a multi-cluster system needs a global load balancer. You don’t want a user request sitting around in some queue when there is spare capacity elsewhere, but you also don’t want to make a habit of sending a request from Singapore to a server in San Francisco. As a rule of thumb, it takes five milliseconds for a request to pass through a time zone. So, sending data from New York to San Francisco takes fifteen milliseconds one way. Given how small latency budgets are, it’s important to run workloads as close to end users as possible. ### 7.3.3 Building for Reliability GPUs are infamous for their high failure rate in production. Every engineer who has done a large-scale training run knows that they need to account for the eventuality that hardware will fail. For example, in their Llama 3 paper, Grattafiori and colleagues revealed that while running 16,000 GPUs for a period of 54 days, the Llama team experienced 419 unexpected interruptions, primarily due to hardware failure. This works out to approximately one failure per 50,000 GPU-hours. 50,000 hours might sound like a long time, but running a single node of eight GPUs for inference for an entire year is over 70,000 GPU-hours. Inference engineers should expect hardware failure. ![Figure 7.12: Root cause of failures when training Llama 3, adapted from “The Llama 3 Herd of Models” (Grattafiori et al., 2024).](https://www.datocms-assets.com/104802/1788123592-inference-engineering-figure-7-12.png) _Figure 7.12: Root cause of failures when training Llama 3, adapted from “The Llama 3 Herd of Models” (Grattafiori et al., 2024)._ GPU health is a node-level concern. When a single GPU fails, other GPUs on the node often fail next or need to be taken offline for maintenance. Proactively noting failures, cordoning nodes, and cycling pods keeps individual clusters healthy. GPU failures aren’t the only thing that can bring down inference. Cloud providers have scheduled maintenance and their own unscheduled downtime. Every layer of infrastructure must be reinforced to provide high reliability. Multi-cloud inference brings two new approaches to high reliability: - **Active-active:** A high‑availability posture where multiple regions or clusters actively serve live traffic at the same time. If any plane fails, traffic seamlessly continues on the others. - **Active-passive:** A failover posture where a “hot standby” cluster or region is kept ready but idle. If the active plane fails, traffic is cut over to the passive plane. When individual clusters, regions, or cloud providers go down, seamlessly failing over to another workload plane keeps reliability high and latency low. ### 7.3.4 Security and Compliance Cloud infrastructure has been a hot topic for security and compliance departments for more than twenty years. For AI models to power mission-critical applications, inference must be both secure and compliant. Security and compliance conversations generally center around three areas: - **User data:** Security and compliance departments want to ensure all data, including user inputs and model outputs, is protected. - **Model weights:** For companies with fine-tuned or proprietary models, the weights are an invaluable trade secret. - **Infrastructure:** GPUs themselves and access to intelligence are both targets for abuse. One of the easiest decisions you can make to improve security is to simply not store user inputs or model outputs. This may not be possible – you might have logging requirements or user agreements to retain usage data for future model training – but if you don’t need to retain user data, you can reduce your attack surface. Securing AI inference workloads and associated data is similar to securing any other containerized workload. Data encryption, container security, network and access controls, and workload isolation, all validated by extensive third-party penetration testing, remain the gold standard. Increasingly, inference engineers need to support applications running in regulated industries and compliance-heavy regions. One place where multi-cloud infrastructure helps is that in order for your application to comply with a certification like SOC 2 Type II or a regulation like HIPAA, your providers generally must also be compliant. In this case, being able to move workloads to compliant providers is useful. Another benefit of multi-cluster infrastructure is running one model across multiple regions. Certain industries and countries have data residency requirements, where user data from their country cannot be processed on servers in a different country. For example, having one cluster in a provider near Toronto and another in a provider near New York lets you keep Canadian data in Canada and American data in the United States while providing minimal latency overhead to users across the geographic region. ## 7.4 Testing and Deployment In addition to any replica-level testing and benchmarking performed while configuring the inference engine, it’s important to test systems end-to-end before deployment. There are several strategies for testing inference: - **Manual testing:** Writing scripts (or clicking buttons) to send synthetic traffic to an inference service. - **Load testing:** Automatically sending a large volume of traffic to test a system’s ability to scale and maintain performance. - **Shadow traffic:** Copying live traffic to test deployments to measure performance under real-world conditions. Testing inference services is expensive. It takes engineering time to configure the tests and measure the results, and it takes GPUs to run inference for the test traffic. To a degree, that’s just the cost of doing business, but think carefully about how to minimize testing expenses. For example, shadow traffic testing could start with copying a random sample of production traffic, followed by a shorter-duration load test. When testing, keep in mind that AI product usage generally fluctuates on daily and weekly cycles. Once you’re confident in the performance and stability of your updated inference system, it’s time to deploy to production. ### 7.4.1 Zero-Downtime Deployment Inference engineers use high-availability deployment strategies to avoid downtime. A traditional high-availability design is a blue-green deployment. In this setup, there are two identical environments: the original blue deployment and a new green deployment running the updated service. Once the green environment is ready, the full traffic load cuts over from the blue to the green environment, with the blue environment staying ready for rollback in case of issues. However, blue-green is not well suited for large scale inference workloads due to the same GPU capacity and cost issues that make large-scale testing difficult. If the blue deployment is using 100 GPUs, the green deployment requires another 100 GPUs before traffic can cut over. Instead, inference engineers can get similar benefits with lower GPU overhead using canary deployments. Inspired by the canaries that were used to detect gas in coal mines, a canary deployment catches errors before they affect large numbers of users. ![Figure 7.13: Iteratively shifting traffic over to the new deployment prevents multiple issues during inference service updates.](https://www.datocms-assets.com/104802/1788123600-inference-engineering-figure-7-13.png) _Figure 7.13: Iteratively shifting traffic over to the new deployment prevents multiple issues during inference service updates._ A canary deployment is a 4-step process: 1. Build a new deployment of the inference service and get it ready to handle incoming requests. 2. Direct a small percentage of the incoming live traffic to the new service. 3. Monitor the new service and ensure it is handling traffic correctly. Revert if there are any issues. 4. Gradually increase traffic, while monitoring for issues, until the new deployment handles 100 percent of traffic. These canary deployments can be rolled out quickly, with just a few minutes of traffic ramp, or ramped slowly to ensure stability at each stage. And with autoscaling, canary deployments don’t increase cost much at scale because reducing traffic to the production system causes it to scale down some replicas. With autoscaling, the new deployment will default to the minimum number of replicas when there is no traffic. Throughout the canary deployment process, ensure that the new deployment has enough active replicas to properly handle requests. Otherwise, users will see a latency spike as their requests are queued until autoscaling completes. ### 7.4.2 Cost Estimation Switching from consuming tokens from a public API to doing your own inference on dedicated GPUs requires changing how you think about cost. Cost on public APIs is simple: a price per million tokens times the number of tokens you use. There are a couple of variables – cache hits versus cache misses for input tokens, discounts for high-volume users – but cost remains a linear function of usage. One motivation for investing the time and effort in inference engineering is to take control of your unit economics and escape per-token pricing. But it’s a difficult mental transition. The blessing and curse of dedicated inference is that cost is now a function of many variables. This is good because it gives you control, but it makes estimation difficult. Factors that affect cost include: - **Batch sizing:** Is the deployment optimized for latency with low batch sizes or throughput with high batch sizes? - **Traffic patterns:** Is traffic consistently saturating active GPUs, or is capacity going spare? - **Sequence lengths:** How many input and output tokens do requests have both on average and in outlier cases? Given this complexity and the difference in cost between input and output tokens, it’s generally more productive to convert your token price into a total cost and compare that to dedicated instead of trying to reverse engineer a per-token price from what you pay for GPUs. ![Figure 7.14: An equation for estimating the total cost of using per-token APIs in a product.](https://www.datocms-assets.com/104802/1788123608-inference-engineering-figure-7-14.png) _Figure 7.14: An equation for estimating the total cost of using per-token APIs in a product._ ![Figure 7.15: An equation for estimating the total cost of using dedicated deployments in a product.](https://www.datocms-assets.com/104802/1788123613-inference-engineering-figure-7-15.png) _Figure 7.15: An equation for estimating the total cost of using dedicated deployments in a product._ Cost estimates should use a long time horizon, ideally at least a week, to smooth out variations in usage. The other factor to consider in dedicated deployments is the cost of engineering time spent building and maintaining inference systems. This investment, while justified in increased reliability, security, and control, should be added to the GPU costs to form a complete picture around total cost of ownership (TCO) for inference. ### 7.4.3 Observability Inference is mission-critical, so it must be monitored like any other mission-critical component of an application, with alerting, logs, and observability built at the right level of abstraction. The first question is what to monitor. Inference observability includes measuring: - **Total volume:** The number of requests that a model deployment is receiving. - **Request and response sizes:** The input and output sequence lengths for the requests being processed. - **Response codes:** The count of 2XX, 4XX, and 5XX response codes issued by the model server. - **Latency:** Metrics like time to first token, tokens per second, and end-to-end latency on a P50, P90, and P99 basis. - **Replica count:** The number of instances actively serving traffic, and the number of instances starting up, if any. - **Utilization:** The amount of utilization across CPU, host memory, GPU, and GPU memory. - **Queue depth:** For systems with asynchronous traffic, the number of requests enqueued and waiting to be processed. These metrics are interdependent. A spike in latency could come from request volume, but it could also come from long input sequences. Seeing these metrics together lets inference engineers understand not only what is happening but also why. When things go wrong, inference engineers need information to fix issues. Logs, both server logs and audit logs showing changes to an inference service, deliver that information in real time. Observability cannot be siloed. When you build observability for inference, build it with deep integration into existing observability and alerting tooling – Grafana, Datadog, PagerDuty, Sentry – to put inference information in context with the rest of the application. ## 7.5 Client Code Inference engineering draws on many technologies, from CUDA to Kubernetes. But there’s one critical area that’s often overlooked when optimizing for latency and building for scale: client code. There are two sides of a call to an inference service: - **Client:** The browser, agent, or application making a request to the inference engine. - **Server:** The inference service that handles the client request and returns the model results. The industry standard for client code is the OpenAI SDK, which supports a wide range of compatible providers in addition to OpenAI’s own models. Popular AI engineering frameworks and libraries like LangChain, Vercel AI SDK, LiteLLM, LlamaIndex, and dozens more can also serve as clients. Whether you’re using an existing library or your own code, there is the potential for latency overhead or throughput bottlenecks. And for real-time applications, you may need a protocol other than HTTP, like WebSockets, to deliver a continuous connection. ![Figure 7.16: On-server inference time is just a fraction of the end-to-end latency for a given request.](https://www.datocms-assets.com/104802/1788123618-inference-engineering-figure-7-16.png) _Figure 7.16: On-server inference time is just a fraction of the end-to-end latency for a given request._ ### 7.5.1 Client Latency Overhead Depending on the client’s internet connection and the protocol used, establishing a session between a client and server takes a few dozen milliseconds. In a high-performance system with a 300-millisecond P95 end-to-end latency SLA, a TLS handshake costs at least ten percent of that latency budget before inference even starts. Future requests from the same client should save time by re-using existing sessions. Session re-use is not a new idea by any means, and tools like the OpenAI SDK provide it silently under the hood. However, when building your own client for non-standard modalities, follow best practices like session re-use. ### 7.5.2 Asynchronous Inference Some systems are built for throughput, not latency. Use cases like bulk document processing and corpus embedding are not latency sensitive, so it makes sense to switch to asynchronous jobs. Asynchronous requests are a “fire and forget” approach to executing inference. Ordinary synchronous inference requests have a timeout, generally of a few minutes, after which the request will fail. Asynchronous jobs fix this by immediately acknowledging the request and later returning the result of the asynchronous job to a webhook supplied in the original request. Asynchronous jobs still have time limits, but these requests are usually measured in hours, not minutes. Along with strong server-side queuing, asynchronous requests make high-throughput, latency-insensitive systems more robust and efficient. ### 7.5.3 Streaming and Protocol Support Streaming makes applications feel instant. For language models, streaming text output over HTTP is sufficient. But for other modalities, especially live voice and video, both input and output streams need to be able to carry more data. ![Figure 7.17: One-time HTTP requests and responses are a good fit for use cases like text chat, but not for continuous streaming.](https://www.datocms-assets.com/104802/1788123623-inference-engineering-figure-7-17.png) _Figure 7.17: One-time HTTP requests and responses are a good fit for use cases like text chat, but not for continuous streaming._ The two most common bi-directional streaming client-server connection protocols are: - **Websockets:** For streaming use cases where strong schema enforcement is not required. - **gRPC:** For well-defined service-to-service communication. WebSockets are useful for transmitting unstructured and real-time data, like audio, where the server receiving the request can parse it and process it downstream. With WebSockets, a server can support up to a fixed, developer-configurable number of clients; when that concurrency is reached, new connections cannot be established and must wait until either a slot is free or another replica scales up. ![Figure 7.18: WebSockets establish a continuous connection for unstructured data like audio streams.](https://www.datocms-assets.com/104802/1788123628-inference-engineering-figure-7-18.png) _Figure 7.18: WebSockets establish a continuous connection for unstructured data like audio streams._ Similar to WebSockets, gRPC enables bi-directional streaming support, but for structured data. Requests transmitted via gRPC must follow a predefined schema, which takes away the load of having to parse the input. This additional validation layer makes gRPC slightly slower than WebSockets. ![Figure 7.19: gRPC establishes a continuous connection for well-defined service-to-service communication.](https://www.datocms-assets.com/104802/1788123633-inference-engineering-figure-7-19.png) _Figure 7.19: gRPC establishes a continuous connection for well-defined service-to-service communication._ ## 7.6 Production Inference with Baseten This book contains everything I’ve learned about inference in four years of working at Baseten. Baseten is an AI infrastructure company founded in 2019. At Baseten, we are focused on highly performant and highly available inference for both open models and custom models. We also offer a platform for pre-training, post-training, and reinforcement learning. We run inference for the world’s fastest-growing startups and most innovative enterprises, including Cursor, World Labs, Notion, OpenEvidence, Clay, Abridge, Gamma, Ambience, Writer, and hundreds more. At Baseten, we focus on four essential pillars to deliver the fastest mission-critical inference: - **Performance:** Consistent low latencies at scale powered by the Baseten Inference Stack. - **Infrastructure:** Reliable multi-cloud deployments, fast and granular autoscaling, and robust security. - **Tooling:** An intuitive and productive developer experience with logging, observability, and programmatic access. - **Applied expertise:** Hands-on-keyboard implementation and assistance from forward deployed engineers. We would be honored to provide fast, reliable inference for your AI-powered products. Also, we are continuously hiring for all roles across engineering, sales, marketing, and operations. To learn more, visit [https://baseten.com/careers](https://baseten.com/careers).