Video generation is the most demanding modality. Whenever possible, these models should be run on Blackwell GPUs (or Rubin, once available). These GPUs offer a high memory capacity for Context Parallelism, fast Tensor Cores for attention computation, and microscaling data formats for more precise quantization.
Architecturally, video generation is similar to image generation, just rendering a full video rather than a single frame from latent space. Following the principle that greater scale unlocks more techniques, video generation uses all the same techniques as image generation plus additional optimizations.
Like image generation, video generation is compute bound and works via iterative denoising over latent space. Video generation models generally take about the same number of denoising steps as image generation models (~50), but each step processes much more data.
As video generation models are compute bound, batching isn’t useful like it is for text generation. Video generation models usually run on full nodes of eight GPUs with a batch size of one: all eight GPUs work together to create one video at a time.
Unlike with batched workloads, where latency-throughput tradeoffs are possible by adjusting the batch size, the only way to improve the throughput and cost of video generation is to make the model itself faster.
Early video generation models were framewise. They generated frames one at a time. This reduced the quality and coherence of the video output. Today, video generation models run denoising steps on the video as a whole in latent space. Where latent space for image generation represents two dimensions (width, height), for video models it represents three (width, height, time).
This means passing huge amounts of data through each attention calculation. For video models, attention is 70 to 80 percent of the compute time, making attention the most important thing to optimize.
6.6.1 Attention Optimization and Quantization
Attention optimization starts with kernel selection. Test FlashAttention, DeepGemm, CuTe, and CUTLASS kernels to see which ones perform best for your model.
Where language models use the KV cache to accelerate attention, video generation models use other caching patterns to attempt to reuse model outputs. Re-using parts of the attention computation can make video generation 30 to 40 percent faster in practice.
Precise methods and algorithms are continuously changing with new research, but there are two fundamental approaches to caching:
- Timestep-based caching: Caching and re-using the outputs of certain timesteps to skip entire steps.
- Transformer-based caching: Caching and re-using hidden states to skip layers within the transformer itself.
Algorithms and implementations range from negligible quality degradation to unusable output – test these strategies carefully before using them in production.
Beyond kernels and caching, the main tool for speeding up attention is quantization.
For bandwidth-constrained language model inference, the benefit of quantization is that it means you have less data to load through memory. For video models, it means you access double the FLOPS by switching to lower-precision Tensor Cores.
However, language model quantization focuses on weights – large linear layers where the impact of quantization is negligible. For video models, quantizing weights still helps, but while these layers take the majority of the memory bandwidth, constraining language models, they’re only a small fraction of the compute time for video models.
Instead, quantization on video models focuses on attention. Attention is the riskiest part of any model to quantize, as errors accumulate over the course of inference. For video models, where there are ~50 steps instead of the thousands of autoregressive iterations in token generation, the risk is slightly lower but still important.
The first method for reducing the quality impact is to use a blockwise quantization and a microscaling data format (MXFP8), both available on Hopper and Blackwell. Microscaling data formats do a better job of preserving outlier values, which have a major impact on attention accuracy.
The most sophisticated approach to attention quantization is selectively quantizing within the model by:
- Step: Keep early steps in FP16 and quantize later steps.
- Layer: Keep first and last layers and quantize hidden layers.
Quantization by step follows the same insight as the classifier-free guidance trick from image generation models: early steps establish the outline of the image, while later steps refine the details. These early steps are more important for prompt adherence and accuracy.
For layers, the first and last layers are more important as they take the input and produce the final output. The hidden layers only perform intermediate calculations which don’t suffer as much from approximation.
By only quantizing less important parts of the video generation process, quality is preserved. These tactics are found in kernels like SageAttention, an 8-bit attention kernel that you can use for quality low-precision attention on video generation models.
6.6.2 Context Parallelism
While video generation models generally run on a full node of eight GPUs, they use Context Parallelism rather than Tensor Parallelism.
Context Parallelism copies the weights onto every GPU. Video models are small enough that replicating the weights eight times takes a meaningful amount of memory but is feasible on B200.
Instead of splitting the model across GPUs, Context Parallelism works by splitting the attention calculation across the GPUs. This is coordinated via a mechanism like ring attention, where each GPU holds a piece of the context and passes intermediate results to the next GPU in the ring.
Attention for transformer models is multi-head, usually with eight or more heads. Attention heads are independent, so they can be run separately with the results combined afterward.
Attention isn’t the only thing that can be parallelized. For example, the latent decoding step using the variational autoencoder takes three to five percent of the total inference time and can be run across GPUs.
These parallelism techniques make AI video feasible. As video sequences get longer and video models get larger, parallelism will continue to be the most critical technique for video generation model inference.