Autoscaling within a single cluster works up to a certain point. But high-volume deployments serving a global user base need thousands of GPUs distributed around the world.
It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fluidly, and moving workloads across clouds is a tedious, error-prone process.
True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling self-healing and global scheduling.
Running true multi-cloud inference unlocks:
- Capacity: Pool capacity from multiple providers for greater and more flexible GPU access.
- Redundancy: Split inference across providers for resiliency against outages.
- Latency: Run inference close to your end users to reduce network latency overhead.
- Compliance: Run inference in compliance with data sovereignty and other regulatory requirements.
Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains:
- Control plane: Handles model deployment and global scaling decisions, receives real-time event streams.
- Workload planes: Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand.
This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected.
7.3.1 GPU Procurement
There are a good number of companies in the business of providing access to GPUs. The three major types are:
- Hyperscalers: Large cloud providers like AWS or GCP.
- Neoclouds: GPU-focused clouds like Coreweave or Nebius.
- Resellers: Secondary markets like SF Compute Company.
Players in this space vary in their capacity, availability, and reliability. You generally pay a premium for hyperscalers and across all providers there is a tradeoff between cost and factors like uptime SLAs, support, regional availability, instance configuration, and cluster sizes.
The first challenge is securing capacity. It is often difficult to get your hands on the GPUs you need, especially the latest hardware. Large clusters are also hard to get, with relatively few players offering blocks of hundreds of nodes.
Many cloud providers allocate the majority of in-demand GPUs to their largest customers on long-term reservations. You may need to work across multiple cloud providers to get the GPUs you need in the right regions.
Cloud GPUs can be procured via three different mechanisms:
- Reserved: Blocks of hundreds or thousands of GPUs are reserved for months or years at discounted rates.
- On-demand: Individual instances are available as needed up to a given quota for a relatively high per-hour cost.
- Spot: Discounted on-demand instances that can be pre-empted at an agreed-upon notice period, often minutes.
Large-scale inference generally uses a blend of GPU sources, with a baseline of low-cost reserved instances and a mix of on-demand and spot for handling peaks in traffic. These GPUs are distributed across multiple clusters worldwide for proximity to end users.
7.3.2 Geo-Aware Load Balancing
Successful AI applications have users all over the world. Just like an individual cluster has a load balancer to ensure that every GPU in the cluster receives the right amount of traffic, a multi-cluster system needs a global load balancer.
You don’t want a user request sitting around in some queue when there is spare capacity elsewhere, but you also don’t want to make a habit of sending a request from Singapore to a server in San Francisco.
As a rule of thumb, it takes five milliseconds for a request to pass through a time zone. So, sending data from New York to San Francisco takes fifteen milliseconds one way. Given how small latency budgets are, it’s important to run workloads as close to end users as possible.
7.3.3 Building for Reliability
GPUs are infamous for their high failure rate in production. Every engineer who has done a large-scale training run knows that they need to account for the eventuality that hardware will fail.
For example, in their Llama 3 paper, Grattafiori and colleagues revealed that while running 16,000 GPUs for a period of 54 days, the Llama team experienced 419 unexpected interruptions, primarily due to hardware failure. This works out to approximately one failure per 50,000 GPU-hours.
50,000 hours might sound like a long time, but running a single node of eight GPUs for inference for an entire year is over 70,000 GPU-hours. Inference engineers should expect hardware failure.
GPU health is a node-level concern. When a single GPU fails, other GPUs on the node often fail next or need to be taken offline for maintenance. Proactively noting failures, cordoning nodes, and cycling pods keeps individual clusters healthy.
GPU failures aren’t the only thing that can bring down inference. Cloud providers have scheduled maintenance and their own unscheduled downtime. Every layer of infrastructure must be reinforced to provide high reliability.
Multi-cloud inference brings two new approaches to high reliability:
- Active-active: A high‑availability posture where multiple regions or clusters actively serve live traffic at the same time. If any plane fails, traffic seamlessly continues on the others.
- Active-passive: A failover posture where a “hot standby” cluster or region is kept ready but idle. If the active plane fails, traffic is cut over to the passive plane.
When individual clusters, regions, or cloud providers go down, seamlessly failing over to another workload plane keeps reliability high and latency low.
7.3.4 Security and Compliance
Cloud infrastructure has been a hot topic for security and compliance departments for more than twenty years. For AI models to power mission-critical applications, inference must be both secure and compliant.
Security and compliance conversations generally center around three areas:
- User data: Security and compliance departments want to ensure all data, including user inputs and model outputs, is protected.
- Model weights: For companies with fine-tuned or proprietary models, the weights are an invaluable trade secret.
- Infrastructure: GPUs themselves and access to intelligence are both targets for abuse.
One of the easiest decisions you can make to improve security is to simply not store user inputs or model outputs. This may not be possible – you might have logging requirements or user agreements to retain usage data for future model training – but if you don’t need to retain user data, you can reduce your attack surface.
Securing AI inference workloads and associated data is similar to securing any other containerized workload. Data encryption, container security, network and access controls, and workload isolation, all validated by extensive third-party penetration testing, remain the gold standard.
Increasingly, inference engineers need to support applications running in regulated industries and compliance-heavy regions. One place where multi-cloud infrastructure helps is that in order for your application to comply with a certification like SOC 2 Type II or a regulation like HIPAA, your providers generally must also be compliant. In this case, being able to move workloads to compliant providers is useful.
Another benefit of multi-cluster infrastructure is running one model across multiple regions. Certain industries and countries have data residency requirements, where user data from their country cannot be processed on servers in a different country.
For example, having one cluster in a provider near Toronto and another in a provider near New York lets you keep Canadian data in Canada and American data in the United States while providing minimal latency overhead to users across the geographic region.