# 7.2 Autoscaling

_Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 7: Production. [Full book index](https://www.baseten.co/inference-engineering/llms.txt)

The goal of autoscaling is to ensure that you always have enough resources to serve all incoming requests while maintaining your latency SLAs without wasting money on idle GPUs.

![Figure 7.3: Without autoscaling, inference systems waste resources during traffic lulls and miss SLAs during traffic spikes.](https://www.datocms-assets.com/104802/1788123541-inference-engineering-figure-7-3.png)
_Figure 7.3: Without autoscaling, inference systems waste resources during traffic lulls and miss SLAs during traffic spikes._

![Figure 7.4: A strong autoscaling system for inference matches resources to demand.](https://www.datocms-assets.com/104802/1788123547-inference-engineering-figure-7-4.png)
_Figure 7.4: A strong autoscaling system for inference matches resources to demand._

Autoscaling systems use Kubernetes, an open-source container orchestration system, along with a cluster-level system for provisioning and deallocating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires.

Kubernetes works by composing a group of hardware resources together into a cluster. This cluster has two types of components:

- **Control plane:** Makes routing and scaling decisions.
- **Worker plane:** Runs the actual containerized applications.

![Figure 7.5: Kubernetes clusters have a single control plane that orchestrates multiple workers.](https://www.datocms-assets.com/104802/1788123555-inference-engineering-figure-7-5.png)
_Figure 7.5: Kubernetes clusters have a single control plane that orchestrates multiple workers._

A Kubernetes cluster can run multiple replicas of multiple models. But how do you decide how many replicas of each model to run? Unless your traffic is unusually consistent, there probably isn’t one number of replicas that perfectly matches your needs.

Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions:

- **Utilization:** Scale up and down based on GPU utilization signals like memory usage or compute usage.
- **Traffic:** Scale up and down based on the number of requests being processed in the system.

Utilization and traffic don’t always match. For example, in LLM prefill, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates.

Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand.

When designing a traffic-based autoscaling system, you want to configure five factors:

- **Min replicas:** What is the minimum number of replicas that stay running, regardless of traffic?
- **Max replicas:** What is the maximum number of replicas that you can allocate when traffic is high?
- **Autoscaling window:** How long is the sliding timeframe that you use to measure traffic and make autoscaling decisions?
- **Scale down delay:** How long after a scale down is suggested do you wait in case of another traffic spike?
- **Concurrency target:** How many requests can each replica handle at once?

The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale down delay prevents premature scaledowns for spikey traffic, but could result in unnecessary spend after traffic truly cools off.

## 7.2.1 Concurrency and Batch Sizing

To properly operate a traffic-based autoscaling system, you need a strong understanding of how much concurrent traffic each instance can handle.

Most model inference services can handle more than one request at a time via batching. There are several approaches to batching:

- **Static batching:** The server waits until the batch is full before starting inference.
- **Dynamic batching:** The server waits until the batch is full or a configured amount of time has passed before starting inference.
- **Continuous batching:** The server continuously runs inference, swapping in requests as slots become available.

Inference engines like vLLM, SGLang, and TensorRT-LLM implement robust continuous batching (or in-flight batching as TensorRT-LLM calls it), where requests are batched at the token level. This minimizes latency relative to static batching.
![Figure 7.6: Static batching sets a fixed batch size and waits for the batch to fill before beginning inference, leading to long wait times for early requests.](https://www.datocms-assets.com/104802/1788123563-inference-engineering-figure-7-6.png)
_Figure 7.6: Static batching sets a fixed batch size and waits for the batch to fill before beginning inference, leading to long wait times for early requests._
![Figure 7.7: Dynamic batching adds a cutoff time after which a batch is run whether or not it is full.](https://www.datocms-assets.com/104802/1788123572-inference-engineering-figure-7-7.png)
_Figure 7.7: Dynamic batching adds a cutoff time after which a batch is run whether or not it is full._

![Figure 7.8: Continuous batching operates at the token level, switching in new requests as old requests finish.](https://www.datocms-assets.com/104802/1788123577-inference-engineering-figure-7-8.png)
_Figure 7.8: Continuous batching operates at the token level, switching in new requests as old requests finish._

Batch sizing trades off latency for throughput. Increasing the batch size will produce more throughput overall, but each user’s latency will get worse. Test performance across multiple batch sizes to find the right fit for your model, instance, latency target, and budget.

This is controlled at the autoscaling configuration level via the concurrency target and at the replica level via the batch size, which should match.

Once every active replica reaches its maximum concurrency, the autoscaling system knows to spin up more replicas. If enough replicas are kicking off half-full batches, it’s time to scale back down.

## 7.2.2 Cold Starts

A cold start is the time it takes to spin up a new replica of a model.

The overall performance of an autoscaling system depends on its cold start speeds. If you can’t spin up replicas fast, it’s hard to confidently scale down, leading to over-provisioning.

There are several factors that affect cold start times:

- **GPU procurement:** How quickly can you add the necessary GPUs to your cluster and allocate them to the model?
- **Image loading:** How quickly can you load the container image onto the newly procured instance?
- **Model loading:** How quickly can you load the model weights into the container?
- **Engine startup:** How quickly can you start your inference engine, including any compilation time?

Each of these factors needs to be optimized separately.

![Figure 7.9: Each step in the cold start process adds to the overall timeline.](https://www.datocms-assets.com/104802/1788123582-inference-engineering-figure-7-9.png)
_Figure 7.9: Each step in the cold start process adds to the overall timeline._

Unless you have a pool of warm nodes that you’re flexing between models, GPU procurement speed is mostly a function of your cloud provider. Section 7.3.1 covers procuring GPUs, and node start time is one of the negotiable factors in a contract.

However, engineers can do a lot on loading images and weights and starting containers and engines.

Loading images and model weights is a function of how quickly you can write gigabytes, often hundreds of gigabytes, of data onto your instance. There are two ways to load images and weights faster: make them smaller or get more bandwidth.

Including only necessary steps and dependencies makes images smaller and faster to build into containers, while quantizing model weights has the additional benefit of making them faster to load during cold starts.

For small models, the strategy used to be baking the weights into the image to simplify caching and loading. However, now that most models have dozens or hundreds of billions of parameters, the weights dwarf the image and are better loaded separately.

Where you load weights from has a massive impact on the bandwidth. If you’re loading from a third party like Hugging Face, you’re limited by their egress speed. And storing your weights in an S3 bucket introduces network latency and data transfer costs.

For loading multi-hundred-billion-parameter models, you need gigabytes per second of bandwidth. The best way to get this is by loading over network within a node from a source cached physically near the GPU instance within the same datacenter.

Inference engines like vLLM and SGLang are fast to start up. But engines like TensorRT-LLM and optimized models with PyTorch have a compilation step that targets the specific hardware resources to build the model inference engine. These compilations often take several minutes.

In these cases, caching built engines massively improves cold start times. Both TensorRT-LLM and PyTorch have image caching mechanisms that make this possible, though you’ll always need to load a cached engine into an instance with exactly the GPU type, CUDA version, and software dependencies as the environment that the engine was built in for it to run properly.

## 7.2.3 Routing, Load Balancing, and Queueing

Once there are multiple replicas online, the system needs to make a decision about which requests to send to which replicas. There are two types of components that make these decisions:

- **Routers:** A router works at the request level to determine the ideal place to send a given request. A router answers “where should this request go?”
- **Load balancers:** A load balancer works at the system level to even out requests between multiple options. A load balancer answers “where could this request go?”

In complex systems, there isn’t just one router and one load balancer. Routing occurs throughout the stack with load balancers injected at key points to keep system-wide performance stable.

Overall, you want to split load equally across replicas. However, routing and load balancing are not as simple as saying, “Well, I have 3 replicas and 12 requests, so let’s put 4 requests on each replica.”

Each request may have a different number of input tokens. If most requests have 100 input tokens, a request with 10,000 will unbalance a simple system. Some requests are also better handled by certain replicas. Examples include:

- **KV cache-aware routing:** Direct a request to a replica that already has a matching prefix in its KV cache.
- **LoRA-aware routing:** Direct a request to a replica that already has the desired LoRA fine-tuned weights in memory.

Intelligent request routing uses information from the inference engine and any orchestrators like NVIDIA Dynamo to route requests based on sequence length, prefix, and LoRA needs.

Load balancing and routing are not enough. When an autoscaling system receives more traffic than it can handle, it needs a way to hold onto requests as it scales up more resources or waits for existing resources to become available.

A queue is the infrastructure primitive for handling this situation. A standard queue is a first-in, first-out system for excess requests, though you can do a more complex implementation like a priority queue to, for example, give paid users priority over free users in high-traffic scenarios.

As new replicas come online, ensure that the queue sees them and requests don’t continue waiting for the existing replicas. Each new replica should immediately be assigned up to its concurrency limit in queued traffic once active.

## 7.2.4 Scale to Zero

Advanced autoscaling systems implement a mechanism for scale to zero, where the system can scale down to zero active replicas if there is no traffic, then scale up automatically when traffic is received.

Scale to zero relies on two prerequisites:

- **Fast cold starts:** As users may be waiting live, cold starts must be as fast as possible.
- **Robust queueing:** The system needs to be able to hold the incoming requests until a replica is live.

Even with these capabilities, scale to zero is not a fit for all workloads.

Scale to zero is great for development, when testing is bursty and latency for the first request is unimportant. And in production, scale to zero is useful for applications that only get traffic periodically, like an agent that’s only accessed during business hours in one country or an offline system designed for daily batch processing jobs.

However, if you’re relying on scale to zero to keep costs low in a latency-sensitive application that gets light, unscheduled traffic, it’s probably a sign that your AI application is not yet ready for dedicated infrastructure and should use pay-per-token APIs until greater scale is reached.

## 7.2.5 Independent Component Scaling

AI applications are increasingly built on multi-model, multi-stage compound AI workloads where inference engineers need to coordinate multiple steps to fulfill a single request.

These steps may have different hardware needs. A voice activity detector model needs a far less powerful GPU than the transcription model it’s chunking data for, while the LLM processing that transcript may need a full node with multiple GPUs. And the scaling parameters for each step in the pipeline also differ.

![Figure 7.10: Independent component scaling gives each model access to appropriate resources and individual scaling.](https://www.datocms-assets.com/104802/1788123587-inference-engineering-figure-7-10.png)
_Figure 7.10: Independent component scaling gives each model access to appropriate resources and individual scaling._

For these pipelines, you need to decompose autoscaling decisions and scale each step individually to right-size resources per step and avoid both bottlenecks and overprovisioning.

However, every model in the pipeline should run in the same cluster. If it takes 10 milliseconds to send a message within a cluster and 50 milliseconds to send messages between clusters, that 40-millisecond difference across a 5-step pipeline would be 20 percent of a one-second latency SLA.