# 5.4 Model Parallelism

_Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 5: Techniques. [Full book index](https://www.baseten.co/inference-engineering/llms.txt)

Every frontier LLM on the market today is too big to fit on a single GPU for batch inference. While GPUs have gotten bigger, so too have models, a trend that does not show signs of reversing.

In FP8, loading a billion parameters of model weights takes roughly a gigabyte of VRAM. For a model like DeepSeek-V3.1, with 671 billion parameters, the model weights alone would cause a single B200 GPU to immediately throw an out-of-memory (OOM) error.

It’s not enough to just barely squeeze the model weights into VRAM. On 4xB200 GPUs, with 720 GB of VRAM, you could theoretically load DeepSeek’s weights. But with no room left over for a KV cache, which often takes up 80 percent or more of the remaining VRAM after weights, four B200 GPUs would not be able to serve DeepSeek with any reasonable sequence length or batch size.

Instead, a full node of eight B200 GPUs is needed to serve real production traffic on a model the size of DeepSeek. You can estimate the minimum number of GPUs required for a model by multiplying the precision, parameter count, and expected KV cache allocation together.

![Figure 5.11: After figuring out how much VRAM inference requires, round up to the next available instance size to determine minimum GPU count.](https://www.datocms-assets.com/104802/1788123437-inference-engineering-figure-5-11.png)
 _Figure 5.11: After figuring out how much VRAM inference requires, round up to the next available instance size to determine minimum GPU count._

In many cases, even for midsize models like GPT OSS, you want to use more than the minimum number of GPUs required to enable larger KV caches and unlock better per-user latency.

However, all of this requires that inference scales efficiently from one GPU to multiple GPUs. The limitation in scaling parallel inference is the communication overhead between GPUs.

Chapter 3 details the different interconnects between GPUs: NVLink and NVSwitch within nodes, InfiniBand between nodes.

While NVLink and InfiniBand offer high bandwidth, they are a fraction of the speed of VRAM. With decode bound on memory bandwidth, multi-GPU inference needs to be carefully designed to avoid bottlenecks in inter-GPU communication. This field of study is called topology-aware parallelism.

There are three primary forms of model parallelism in inference:

- **Pipeline Parallelism (PP):** Splits the layers of the model across GPUs.
- **Tensor Parallelism (TP):** Splits the tensors within each layer across GPUs.
- **Expert Parallelism (EP):** Shards entire experts from MoE models across different GPUs.

Each form of parallelism has its own tradeoffs:

| Method | Mechanism                                                               | Drawback                                                                        |
| :----- | :---------------------------------------------------------------------- | :------------------------------------------------------------------------------ |
| PP     | Each GPU handles a stage of the forward and backward pass.              | Not recommended due to poor latency and utilization from step-by-step pipeline. |
| TP     | Compute-heavy operations like matmuls are split across GPUs.            | Requires synchronization across GPUs, not suitable for multi-node.              |
| EP     | Each expert lives within a single GPU, making in-expert inference fast. | Requires routing between GPUs to reach multiple experts, better for throughput. |

Tensor Parallelism is generally best for low-latency model inference within a single node, while Expert Parallelism improves throughput for MoE LLMs. Pipeline Parallelism is only used for multi-node inference.

Additionally, data parallelism strategies like Context Parallelism split computation across devices. These strategies are rare in LLM inference but essential for video generation (section 6.6).

## 5.4.1 Tensor Parallelism for Lower Latency

Tensor Parallelism should be your default strategy for multi-GPU model inference. It supports both dense models like Llama 405B and the MoE models that currently dominate the open model landscape.

![Figure 5.12: Tensor Parallelism splits weights across GPUs, effectively sharing VRAM resources to run large models fast.](https://www.datocms-assets.com/104802/1788123445-inference-engineering-figure-5-12.png)
_Figure 5.12: Tensor Parallelism splits weights across GPUs, effectively sharing VRAM resources to run large models fast._

TP works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs. For each layer, the expense of reading from weights memory and executing matrix multiplication is shared across the GPUs.

![Figure 5.13: For Mixture of Experts models, each expert runs across multiple GPUs with Tensor Parallelism.](https://www.datocms-assets.com/104802/1788123450-inference-engineering-figure-5-13.png)
_Figure 5.13: For Mixture of Experts models, each expert runs across multiple GPUs with Tensor Parallelism._

However, the results of each layer need to be communicated in an all-reduce fashion into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized.

Increasing Tensor Parallelism improves TPS on a per-user basis (assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass, which is the case for most frontier models).

## 5.4.2 Expert Parallelism for Higher Throughput

Expert Parallelism neatly divides experts across GPUs. In a model with 128 experts served in EP8 across eight GPUs, each GPU will host 16 full experts.

![Figure 5.14: Expert Parallelism runs each expert within a single GPU, with GPUs each hosting multiple experts.](https://www.datocms-assets.com/104802/1788123455-inference-engineering-figure-5-14.png)
_Figure 5.14: Expert Parallelism runs each expert within a single GPU, with GPUs each hosting multiple experts._

EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens.

Many deployments use a mix of TP and EP to achieve both benefits.

![Figure 5.15: This deployment uses TP for attention and EP for the sparse MoE layer.](https://www.datocms-assets.com/104802/1788123225-inference-engineering-figure-2-9.png)
_Figure 5.15: This deployment uses TP for attention and EP for the sparse MoE layer._

Expert Parallelism requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model. Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer.

Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth.

## 5.4.3 Multi-Node Inference

If you’re serving a huge model at high precision, supporting multi-million-token input sequences, or just trying to run inference as fast as possible, you might need more than eight GPUs.

![Figure 5.16: InfiniBand enables multi-node inference across more than eight GPUs via high-bandwidth node-to-node interconnect.](https://www.datocms-assets.com/104802/1788123318-inference-engineering-figure-3-4.png)
_Figure 5.16: InfiniBand enables multi-node inference across more than eight GPUs via high-bandwidth node-to-node interconnect._

GPUs are designed to work together across nodes, and multi-node training has been the standard for years to develop frontier models. But multi-node inference introduces new challenges:

- **Infrastructure:** How do you reliably provision two or more interconnected GPU nodes and build abstractions across cloud providers (chapter 7)?
- **Parallelism:** How do you effectively communicate over InfiniBand, which is much slower than NVLink?

InfiniBand introduces a new wrinkle to topology-aware parallelism. Tensor Parallelism generally requires too much communication across GPUs to be a good fit for multi-node inference. Instead, you have two options that work well over InfiniBand:

1. For dense models, use Tensor Parallelism within each node and Pipeline Parallelism between nodes (e.g., TP8PP2).
2. For MoE models, you can also try Expert Parallelism (e.g., EP16) as it has a lower communication overhead than Tensor Parallelism.

For MoE models, TP8PP2 will generally offer lower latency per user and EP16 will yield higher overall system throughput.

Unless your model and KV cache are so large as to require multi-node inference, it probably isn’t the best use of the extra hardware. You’re often better off using the extra nodes for horizontal scaling across replicas, or for disaggregated serving.