In addition to any replica-level testing and benchmarking performed while configuring the inference engine, it’s important to test systems end-to-end before deployment.
There are several strategies for testing inference:
- Manual testing: Writing scripts (or clicking buttons) to send synthetic traffic to an inference service.
- Load testing: Automatically sending a large volume of traffic to test a system’s ability to scale and maintain performance.
- Shadow traffic: Copying live traffic to test deployments to measure performance under real-world conditions.
Testing inference services is expensive. It takes engineering time to configure the tests and measure the results, and it takes GPUs to run inference for the test traffic. To a degree, that’s just the cost of doing business, but think carefully about how to minimize testing expenses. For example, shadow traffic testing could start with copying a random sample of production traffic, followed by a shorter-duration load test.
When testing, keep in mind that AI product usage generally fluctuates on daily and weekly cycles.
Once you’re confident in the performance and stability of your updated inference system, it’s time to deploy to production.
7.4.1 Zero-Downtime Deployment
Inference engineers use high-availability deployment strategies to avoid downtime.
A traditional high-availability design is a blue-green deployment. In this setup, there are two identical environments: the original blue deployment and a new green deployment running the updated service. Once the green environment is ready, the full traffic load cuts over from the blue to the green environment, with the blue environment staying ready for rollback in case of issues.
However, blue-green is not well suited for large scale inference workloads due to the same GPU capacity and cost issues that make large-scale testing difficult. If the blue deployment is using 100 GPUs, the green deployment requires another 100 GPUs before traffic can cut over.
Instead, inference engineers can get similar benefits with lower GPU overhead using canary deployments. Inspired by the canaries that were used to detect gas in coal mines, a canary deployment catches errors before they affect large numbers of users.
A canary deployment is a 4-step process:
- Build a new deployment of the inference service and get it ready to handle incoming requests.
- Direct a small percentage of the incoming live traffic to the new service.
- Monitor the new service and ensure it is handling traffic correctly. Revert if there are any issues.
- Gradually increase traffic, while monitoring for issues, until the new deployment handles 100 percent of traffic.
These canary deployments can be rolled out quickly, with just a few minutes of traffic ramp, or ramped slowly to ensure stability at each stage. And with autoscaling, canary deployments don’t increase cost much at scale because reducing traffic to the production system causes it to scale down some replicas.
With autoscaling, the new deployment will default to the minimum number of replicas when there is no traffic. Throughout the canary deployment process, ensure that the new deployment has enough active replicas to properly handle requests. Otherwise, users will see a latency spike as their requests are queued until autoscaling completes.
7.4.2 Cost Estimation
Switching from consuming tokens from a public API to doing your own inference on dedicated GPUs requires changing how you think about cost.
Cost on public APIs is simple: a price per million tokens times the number of tokens you use. There are a couple of variables – cache hits versus cache misses for input tokens, discounts for high-volume users – but cost remains a linear function of usage.
One motivation for investing the time and effort in inference engineering is to take control of your unit economics and escape per-token pricing. But it’s a difficult mental transition.
The blessing and curse of dedicated inference is that cost is now a function of many variables. This is good because it gives you control, but it makes estimation difficult. Factors that affect cost include:
- Batch sizing: Is the deployment optimized for latency with low batch sizes or throughput with high batch sizes?
- Traffic patterns: Is traffic consistently saturating active GPUs, or is capacity going spare?
- Sequence lengths: How many input and output tokens do requests have both on average and in outlier cases?
Given this complexity and the difference in cost between input and output tokens, it’s generally more productive to convert your token price into a total cost and compare that to dedicated instead of trying to reverse engineer a per-token price from what you pay for GPUs.
Cost estimates should use a long time horizon, ideally at least a week, to smooth out variations in usage.
The other factor to consider in dedicated deployments is the cost of engineering time spent building and maintaining inference systems. This investment, while justified in increased reliability, security, and control, should be added to the GPU costs to form a complete picture around total cost of ownership (TCO) for inference.
7.4.3 Observability
Inference is mission-critical, so it must be monitored like any other mission-critical component of an application, with alerting, logs, and observability built at the right level of abstraction.
The first question is what to monitor. Inference observability includes measuring:
- Total volume: The number of requests that a model deployment is receiving.
- Request and response sizes: The input and output sequence lengths for the requests being processed.
- Response codes: The count of 2XX, 4XX, and 5XX response codes issued by the model server.
- Latency: Metrics like time to first token, tokens per second, and end-to-end latency on a P50, P90, and P99 basis.
- Replica count: The number of instances actively serving traffic, and the number of instances starting up, if any.
- Utilization: The amount of utilization across CPU, host memory, GPU, and GPU memory.
- Queue depth: For systems with asynchronous traffic, the number of requests enqueued and waiting to be processed.
These metrics are interdependent. A spike in latency could come from request volume, but it could also come from long input sequences. Seeing these metrics together lets inference engineers understand not only what is happening but also why.
When things go wrong, inference engineers need information to fix issues. Logs, both server logs and audit logs showing changes to an inference service, deliver that information in real time.
Observability cannot be siloed. When you build observability for inference, build it with deep integration into existing observability and alerting tooling – Grafana, Datadog, PagerDuty, Sentry – to put inference information in context with the rest of the application.