# Chapter 6: Modalities _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) The modality of a model describes what types of input it accepts and what types of output it creates. Chapters 1 through 5 focus on inference engineering for LLMs, which take text as input and produce text as output. This chapter expands the discussion to more modalities. Generative AI models offer a rich array of modalities, including: | Input | Output | Category | | :------------------- | :------------ | :--------------------- | | Text and image/video | Text | Vision language | | Text or image/video | Vector | Embedding | | Audio (voice) | Text | Transcription | | Text | Audio (voice) | Speech synthesis | | Text | Audio (music) | Music generation | | Audio (voice) | Audio (voice) | Speech-to-speech | | Text and/or image | 3D model | Generative CAD | | Text and/or image | Image/Video | Image/video generation | | Image/video | Text | Captioning | | Image/video | Mask | Segmentation | | Text and image | Image | Image editing | Fortunately, while there are many modalities, there are just two broad archetypes of generative AI models as outlined in chapter 2: - **Autoregressive token generation:** Start from a tokenized sequence and predict the most likely next token. - **Iterative denoising:** Start from random noise and refine toward the most likely output. LLMs are the most famous autoregressive transformers models for token generation, but far from the only ones. Vision language models, text and multimedia embedding models, automatic speech recognition (ASR) models, text-to-speech (TTS) models, and many others rely on similar architectures. Many of the same inference engines and techniques used for LLMs also apply to these related modalities. Image and video generation models instead rely on iterative denoising, though increasingly hybrid diffusion transformer models are setting the frontier in quality. While a number of the same philosophies from kernel selection to parameter tuning also apply to image model optimization, the details end up quite different. For each new modality, you also need to adjust the way you think about and measure latency, throughput, and quality. For example, a single token of audio output from a TTS model isn’t particularly useful; instead of TTFT, measure the time to first word or time to first sentence. This chapter discusses inference engineering for six common modalities beyond LLMs, with special attention to the different considerations for each modality. ## 6.1 Vision Language Models Vision language models (VLMs) take one or more images or videos as input along with a text prompt and generate a text response. ![Figure 6.1: VLMs add image and video understanding to LLMs.](https://www.datocms-assets.com/104802/1788123468-inference-engineering-figure-6-1.png) _Figure 6.1: VLMs add image and video understanding to LLMs._ A vision language model usually consists of two modules: - **LLM:** A standard large language model. - **Vision encoder:** A small model that takes raw images and videos as input and converts them into image tokens. The language model is much larger than the vision encoder. For example, in Mistral Large 3, the vision encoder is just two billion parameters compared to the 673B-parameter LLM. While the vision encoder is small by parameter count, it is critical for inference. VLMs use varied architectures and implementations for vision encoders, so runtime support is somewhat more fragmented for vision language models. This fragmentation increases the importance of vLLM and SGLang for serving vision language models. As a rule of thumb, sending a high-resolution input image to a VLM adds about a thousand visual tokens to the input sequence. While at a very high level image tokens are similar to regular tokens, they add up quickly. Across VLMs, the primary challenge in inference optimization is handling the longer input sequence and larger KV cache. This adds wrinkles at both phases of inference: - **Prefill:** Images are patched, embedded, tokenized, and fed into prefill as part of the input sequence. - **Decode:** Same mechanics, longer context, and some models add attention variants for the image tokens. Every technique from the previous chapter is useful in addressing this challenge: - **Quantization:** KV cache quantization reduces the memory bandwidth and storage overhead for longer sequences. - **Speculation:** Decode for VLMs matches LLMs and can be accelerated with speculation, especially EAGLE. - **Prefix caching:** Re-use KV cache for images in multi-turn chats and repeated queries. - **Parallelism:** Use Tensor Parallelism for fast inference while accessing more VRAM for large models and long contexts. - **Disaggregation:** Move prefill to specialized and independently scaling workers to handle long sequences. In addition to these techniques, VLMs introduce a new quality-speed tradeoff: downsampling. Images and videos can be converted into visual tokens at various resolutions. A high-resolution representation takes about four times more tokens than a low-resolution image, but provides more detailed information. Downsampling generally isn’t needed for single-image inputs, but it may be needed when passing in multiple images or video clips. ### 6.1.1 Video Processing for Vision Language Models A video is more than the sum of its frames. Videos may contain audio (though many VLMs cannot process audio, which must be transcribed separately and added into the prompt) and their frames express motion of objects through space that is lost when looking at static images. VLMs are trained on video clips to understand that time dimension. High-quality inference requires processing the entire video clip in a single call to the model. One second of cinematic video contains 24 frames. Each frame is an image. If a high-definition input image takes about 1,000 tokens to represent, then a four-second video clip produces an input sequence of nearly 100,000 tokens. In reality, video inputs don’t generate quite this long of an input sequence – downsampling is practically obligatory. Reducing the resolution and frame rate makes it possible to evaluate an entire clip in a single inference request, though video understanding models are still only capable of taking very short clips. After the video is tokenized and encoded, inference is similar to working with images, just with much longer context. Prefix caching, KV cache offloading, and optimized attention implementations are of even greater importance for these input sequences of tens of thousands of tokens. ### 6.1.2 Omni-Modal Models Vision language models are an important part of a trend toward “omni” models that accept multiple types of input and produce multiple types of output. There are pros and cons to omni models – their blend of modalities provides unique capabilities, but smaller specialized models are often faster and more accurate within specific domains. For example, many VLMs have text recognition capabilities trained into their image input processing. However, these capabilities lag behind dedicated optical character recognition (OCR) models that are generally a fraction of the size. Running production inference on VLMs often involves coordinating a pipeline of multiple models and pre-processing steps. You might have individual preprocessors for extracting data from PDFs, reading text from images via OCR, or transcribing audio from a video. Each component in the pipeline must be individually optimized for speed and should scale independently to avoid bottlenecks. ## 6.2 Embedding Models An embedding model transforms a variable-length chunk of text – or another modality of input like an image – into a fixed-length vector representation that captures the semantic meaning of the input. ![Figure 6.2: Embedding models convert unstructured input data into vectors that encode semantic meaning.](https://www.datocms-assets.com/104802/1788123473-inference-engineering-figure-6-2.png) _Figure 6.2: Embedding models convert unstructured input data into vectors that encode semantic meaning._ By encoding content into this shared semantic vector space, you can compare distance between items with simple math. Embedding models (along with vector databases) are used to build agent memory, RAG, search, and recommendation systems. To support these use cases, embedding model inference workloads have two different traffic profiles: 1. **High-throughput backfills:** Bulk operations like indexing millions of documents, updating product catalogs, or even preparing data for LLM pre-training. 2. **Low-latency lookups:** Individual user-facing queries for search, retrieval, or recommendation, where every millisecond affects user experience. Inference engineering for embedding models starts with clarifying which profile you need to serve. If you need to do both and have enough traffic to justify the cost, it’s better to build a separate system for each type of usage. ### 6.2.1 Embedding Model Architecture There are tens of thousands of embedding models on Hugging Face, but they all use one of two transformers-based architectures: - **BERT-style models:** Encoder-only neural networks, usually <1B parameters, originally built for masked token prediction. - **LLM-based models:** Modern language models, generally <=8B parameters, repurposed to generate embeddings. Today, LLM-based embedding models offer substantially greater capabilities, though BERT-style models are still used for simple latency-sensitive tasks like classification. Embedding models introduce their own speed/quality tradeoff in embedding dimensionality, or the size of their output vectors. An embedding vector contains a few hundred to a few thousand values, with longer vectors encoding more information. Modern embedding models use Matryoshka representations to unlock dynamic tradeoffs between embedding dimensionality and quality while retaining more information on shorter vectors. Dimensionality doesn’t materially affect inference time but does affect the storage, retrieval, and similarity computation time within a system. In most cases, vectors from one embedding model cannot be meaningfully compared to vectors from another embedding model, even if they are the same length, as they encode inputs into different semantic spaces. ### 6.2.2 Embedding Model Inference For embedding models with LLM backbones, like Qwen 3 Embed 8B, inference optimization shares common tools and techniques with other high-volume, low-latency deployments of smaller LLMs. There are multiple runtimes for text embedding models: vLLM, SGLang, Infinity, TEI (Text Embedding Inference by Hugging Face). But the best performance comes from adapting TensorRT-LLM to run these models. ![Figure 6.3: A high-performance embedding inference pipeline adds parallel tokenization and batch management in front of an optimized inference engine.](https://www.datocms-assets.com/104802/1788123478-inference-engineering-figure-6-3.png) _Figure 6.3: A high-performance embedding inference pipeline adds parallel tokenization and batch management in front of an optimized inference engine._ TensorRT-LLM brings an optimized XQA kernel for fast attention and kernel fusion techniques to reduce memory access overhead. For supported models, TensorRT-LLM is the most performant inference engine for both latency and throughput. Further gains come from quantization. While smaller models are more likely to lose quality from quantization, FP8 quantization for embedding model weights offers improved performance with minimal quality loss. The easiest way to check embedding model quality post-quantization is to run the same inputs through both the original and the quantized model, then check the cosine similarity of the output vectors. A cosine similarity of one hundred percent means the vectors are identical; you’ll want to see at least 99 percent similarity to have confidence in the quantization. As embedding models process tokens in parallel, prefix caching and disaggregation aren’t relevant optimizations. And given these models’ small size, parallelism across multiple GPUs is not effective. Instead, high-traffic deployments should scale horizontally, with each GPU as its own replica. In high-traffic deployments of embedding models, batching and queueing play an important role in performance. Embedding models offer much higher batch sizes than other models. A single request may batch dozens or hundreds of text inputs together in a list, and many requests can run in parallel on a single GPU as even the most demanding embedding models are relatively small and fast. Whether you’re performing a large backfill or handling a surge in usage, traffic can exceed even the large batch sizes offered by embedding models. In these cases, a robust queuing system is essential infrastructure for supporting embedding model inference. ## 6.3 ASR Models Automatic speech recognition (ASR) models take audio as input and produce text as output, powering transcription and dictation apps. The most popular open ASR model is Whisper, which was released by OpenAI. Whisper supports dozens of languages with accurate transcription. ![Figure 6.4: ASR models transcribe input audio into text.](https://www.datocms-assets.com/104802/1788123486-inference-engineering-figure-6-4.png) _Figure 6.4: ASR models transcribe input audio into text._ Whisper comes in various sizes, but the largest and highest-quality Whisper model is just 1.55B parameters. Whisper runs extremely fast on fractions of large GPUs like H100 via Multi-Instance GPUs (MIGs). While various other sizes, variants, distillations, and quantizations exist, in practice it’s possible to satisfy most latency budgets with the highest-quality models: Whisper 3 Large and Whisper 3 Turbo. Whisper is an encoder-decoder model: - **Encoder:** Takes a processed audio waveform (log-Mel spectrogram) as input and encodes it into audio features. - **Decoder:** Takes these encoded audio features and converts them into text tokens. The overwhelming majority of inference time is spent on the decoder, which is an autoregressive transformer model very similar in architecture to an LLM. Fortunately, there are excellent tools for optimizing the main bottleneck. The main tool for performance optimization on the decoder side is TensorRT-LLM. With TensorRT-LLM, you can get in-flight batching for the decoder and an optimized C++ runtime with highly efficient CUDA kernels. TensorRT-LLM works especially well with recent architectures like Hopper and Blackwell, making MIGs an even better option for ASR inference. ### 6.3.1 Single-Chunk Latency Optimization One use case for Whisper is real-time transcription, like in a dictation app or voice agent. For live Whisper, look at round-trip time for a single chunk of audio to be transcribed. A great target to aim for is 200 milliseconds, which is the average human reaction time. With Whisper running on an optimized TensorRT-LLM inference engine, there isn’t a lot of work to do on the runtime level to improve performance. Instead, most gains for real-time Whisper come from orchestration and infrastructure. The biggest upgrade in product experience for ASR is streaming, which is implemented at the API server layer rather than the model runtime layer. By establishing a WebSocket connection (section 7.5.3) and streaming audio continuously in and text continuously out, products can transcribe in real time rather than transcribing pre-recorded audio. At the ASR runtime layer, nothing changes. Instead, a streaming implementation for transcription uses a Voice Activity Detection (VAD) model to monitor the incoming stream and segment it into discrete chunks for the ASR model to process. Inference is run on the chunks as normal, and the text results are streamed back via the WebSocket. This setup can handle several concurrent streams, and it has the advantage of keeping transcription sequential. When each chunk is processed on the same GPU, you can use the output sequence of the previous chunk as the prefix for the next chunk, improving transcription quality. ### 6.3.2 Long File Latency Optimization One limitation of the Whisper model is that it can only support 30-second chunks. Transcribing long files, like hour-long podcasts, requires a different set of optimizations. Measure the performance of long file transcription with the confusingly named Real-Time Factor (RTF). If the world’s fastest typist could manually transcribe an hour of audio in 30 minutes, they would have an RTF of 2X. With a Whisper deployment optimized for long files, you can transcribe an hour of audio in less than four seconds, for an RTF of 1000X. Fast transcription for long files requires a multi-step pipeline. The first step is again a VAD model, this time running on its own dedicated hardware. The model is used to remove silence and chunk out meaningful audio segments rather than splitting by time intervals, which runs the risk of cutting words in half. Then, the chunks can be processed in parallel. Ideally, you use multiple GPUs (or multiple MIGs) to process more audio chunks at once. RTF improves roughly linearly with the number of GPUs used. Each GPU processes multiple chunks at once with in-flight batching for high utilization. Finally, the chunked transcripts are stitched back together by timestamp. ![Figure 6.5: A two-stage pipeline for long audio file transcription parallelizes chunk transcription to improve end-to-end request time.](https://www.datocms-assets.com/104802/1788123491-inference-engineering-figure-6-5.png) _Figure 6.5: A two-stage pipeline for long audio file transcription parallelizes chunk transcription to improve end-to-end request time._ Parallelizing chunk transcription removes the ability to use the previous sequence as a prefix for the next sequence. But there are other quality improvement techniques that more than make up for this. With ASR output, you can automatically detect hallucinations like repeated words and phrases by measuring the compression ratio and words per minute of the output. When a chunk has an issue, you can: 1. Re-run the chunk with a higher temperature. This is counterintuitive – higher temperatures generally produce more hallucinations – but the intention is to break cycles of repeated words and generate a different output. 2. Re-chunk the entire audio, or a segment of the audio, into smaller chunks and re-run the transcription. In practice, these techniques obviate the need for passing a previous sequence as a prefix, unlocking highly efficient and accurate parallel transcription for long files. ### 6.3.3 Diarization Diarization, or annotating a transcript with who is speaking when, is an adjacent problem to transcription. Diarization models categorize audio by voice feature, then segment and cluster across the file to timestamp changes in speaker. Diarization models are a completely different class of model. Where Whisper is an encoder-decoder transformers model, diarization systems like pyannote audio are pipelines of classic ML models. A diarization pipeline contains models for segmentation, embedding, and clustering. To optimize diarization, you have to run each model fast and orchestrate the whole pipeline efficiently. As diarization is an ML pipeline, you can use tools like PyTorch and pyannote along with optimizations like Torch compilation to improve its performance. In practice, even highly optimized implementations of diarization take at least twice as long to process an audio file versus transcription. ## 6.4 TTS Models Text-to-speech (TTS) models, also called speech synthesis models, take text as input and produce audio as output, specifically generating speech. In 2025, open models like Orpheus TTS introduced extremely lifelike speech synthesis to the open model ecosystem. Many companies fine-tuned Orpheus for increased vocal quality and product-specific voices, leading to high adoption of open models in the voice AI space. ![Figure 6.6: Text to speech models synthesize input text into audio.](https://www.datocms-assets.com/104802/1788123500-inference-engineering-figure-6-6.png) _Figure 6.6: Text to speech models synthesize input text into audio._ Modern TTS models are fine-tuned LLMs. Orpheus TTS, for example, is derived from Llama 3.2 3B. This means that many of the same runtime and performance optimizations developed for LLMs apply to speech synthesis models. TTS models have a small parameter count – Orpheus TTS at three billion is on the larger end – meaning that like ASR models, MIGs on H100s are highly efficient and performant options for inference. Unlike ASR models, which generally run in FP16, TTS model weights and KV cache can be quantized to FP8 for better performance in addition to the optimized kernels and in-flight batching introduced by the TensorRT-LLM inference engine. TTS models with LLM backbones are trained by expanding the vocabulary size of the LLM with tens of thousands of encoded audio tokens. Then, the models are trained on pairs of text inputs with tokenized audio outputs. This means that to use TTS models in practice, you also need an audio decoder that takes the audio output tokens and converts them into a waveform. This audio decoding process adds a potential bottleneck to inference. The audio decoder should be implemented using PyTorch and compiled for efficient operation on the target GPU and should use dynamic batching with a short timeout (e.g., 15 milliseconds). In-flight batching is not possible for the audio decoder. TTS model performance is measured with somewhat different metrics than LLMs. The key metrics are: - **TTFB:** Time to first byte (TTFB) is the equivalent of TTFT for speech synthesis. - **Time to first sentence:** Instead of TTFB, a more user-oriented latency metric is the time to generate the first meaningful phrase or sentence. - **TPS:** Like an LLM, the TTS model generates tokens, so decode speed can be measured in tokens per second. Like with TTFT on LLMs, the goal is to minimize TTFB for speech synthesis. For Orpheus, it’s possible to get as low as 150 milliseconds on a single H100. However, there are different goals for TPS on speech synthesis models. The tokens that the model generates are converted to audio waveforms. Depending on the model, it might take 80 to 100 tokens per second to generate audio in real time. Beyond that level, there isn’t any benefit to generating additional tokens per second. Instead, performance enhancements are used to scale throughput in terms of the number of concurrent real-time outputs the model can create. If a single GPU can support many concurrent users, the per-user cost of speech synthesis drops dramatically. ### 6.4.1 Streaming Real-Time Text to Speech Most TTS tasks call for real-time speech synthesis. Like with ASR models, the performance gains for real-time systems come less from the runtime layer – which has already been optimized with TensorRT-LLM, quantization, and a compiled SNAC decoder – and instead from infrastructure. Again, streaming over WebSockets is the biggest unlock for performance versus sending text and receiving audio in discrete chunks. After testing the inference engine to determine how many concurrent real-time streams can be generated, set the same batch size and active WebSocket count to keep usage high but stable. TTS models are rarely used outside of real-time applications. However, if you do end up with a batch use case like backfilling a large corpus of documents to audio for improved accessibility, note that TTS models don’t do well with long inputs, speech starts to degrade after 30 seconds or so. ### 6.4.2 Speech-to-Speech Models One exciting area of research is speech-to-speech models, or models that take audio as input and generate audio as output. Today, most voice systems use a cascading approach, where an ASR model, LLM, and TTS model work in a pipeline to listen, think, and respond to users. These pipelines also employ auxiliary components like VAD and embedding models to facilitate natural conversation and add context. ![Figure 6.7: Most voice-based applications use a cascading approach with a multi-model pipeline.](https://www.datocms-assets.com/104802/1788123504-inference-engineering-figure-6-7.png) _Figure 6.7: Most voice-based applications use a cascading approach with a multi-model pipeline._ Speech-to-speech models, like OpenAI’s gpt-realtime, augment a core LLM with audio consumption and production capabilities, effectively unifying the pipeline in a single model. This is possible thanks to ASR, LLM, and TTS sharing such similar architectures, especially on the decoder. At the time of publication, there are no commercially viable open speech-to-speech models, and closed options like gpt-realtime are significantly less capable and more expensive than cascading multi-model setups. However, research in this space is robust, and this emerging modality will soon require its own flavor of inference engineering. ## 6.5 Image Generation Models ![Figure 6.8: Image generation models may accept both text and reference images to create new output images.](https://www.datocms-assets.com/104802/1788123509-inference-engineering-figure-6-8.png) _Figure 6.8: Image generation models may accept both text and reference images to create new output images._ Working with image and video generation models is entirely different from working with large language models on a few axes. The first is architecture. While some recent models like HunyuanImage-3.0 more closely resemble LLMs, most image and video generation models are iterative denoisers, not autoregressive token generators. Image generation models are pipelines with multiple small models working together in latent space rather than the uniform decoder architecture of an LLM. As such, the tooling is different. At the time of publication, SGLang Diffusion and vLLM Omni are brand new. Most image and video generation model inference is implemented lower in the stack, working with PyTorch or TensorRT directly. The constraints are different too. Image generation models are ten to twenty times smaller than frontier language models, and inference is constrained on compute, not bandwidth. But perhaps the most significant difference is that image and video generation models offer more direct quality to speed tradeoffs. Evaluating image model output quality is difficult to do programmatically. Automatic pipelines using vision language models give directional signal at best and may diverge from human preferences. The human eye is mysterious, and most image quality evals work by asking humans to pick among thousands of images to aggregate vibes and preferences into quality benchmarks. ### 6.5.1 Image Generation Kernel Optimization When you read a model card for an image generation model from its repository, inference examples generally use the `diffusers` library with very few optimizations. In fact, while image generation is theoretically compute bound, you often need to select memory-efficient kernels and use kernel fusion to even reach that bottleneck. High-performance image model inference uses one of three libraries: - **SGLang Diffusion:** Performant inference engines built for popular image and video generation architectures. - **TensorRT:** High-quality black-box implementations of popular models with NVIDIA’s in-house kernels. - **PyTorch:** Careful kernel selection and fusion yields control, flexibility, and improved high-end performance. If you want something that works well and you want it now, just use the SGLang Diffusion or TensorRT implementation of a model. But with PyTorch, there’s an opportunity for advanced inference engineers to do deep customization. The most essential kernel is the attention kernel. Many image generation models use FlashAttention 2 out of the box, but FlashAttention 3 and 4 yield better performance on Hopper and Blackwell GPUs, respectively. There is a whole barrage of smaller kernels, especially normalization functions like RMSNorm, that are good candidates for fusion to ensure efficient memory usage. Then, GEMM kernels matter for compute-bound inference. GEMM kernels apply to linear layers, and are generally safe to quantize into 8-bit floating point formats to access two times higher FLOPS on Tensor Cores. Kernels from CuTe, CUTLASS, or DeepGEMM may prove most efficient on a model-by-model basis. Torch compilation includes automatic kernel fusion with a plugin system for inserting manually selected kernels, and the resulting engine can be cached for faster load times on node startup (which is important because compilation takes several minutes). Like most high-performance engines, Torch compilation targets the specific GPU model and architecture performing the compilation – if you want to run the model on a B200, do the compilation on a B200. ### 6.5.2 One Weird Trick for Faster Image Generation Kernel selection and Torch compilation are all bona fide inference optimization techniques. But the world of inference optimization has fun hacks as well, and here’s one of them. ![Figure 6.9: Recall that diffusion is a step-by-step process and that the general outline of the image is established in early steps.](https://www.datocms-assets.com/104802/1788123234-inference-engineering-figure-2-10.png) _Figure 6.9: Recall that diffusion is a step-by-step process and that the general outline of the image is established in early steps._ Image generation time tracks linearly with step count. That’s why few-step models and latent consistency models are so much faster than full 50-step models. But reducing step count may reduce image quality below an acceptable threshold. Each pass through the denoising model is run at a batch size of two as each step includes a pass with and without prompt guidance. As a refresher, the guidance parameter controls how much the prompt-guided image is weighted when combining the two iterations generated on each step. If the guidance is zero, the prompt-guided image does not need to be generated. After the first few steps, the basic outline of the image is in place, and the rest of the steps are for filling in the details. Thus, prompt adherence is more important in early steps that affect the broad strokes of the image – the model is not going to change its mind on later steps and generate a dog when it is in the middle of generating a cat. If you turn off guidance partway through the image generation, you save passes through the denoiser without reducing step count. If guidance is skipped for the last 20 steps of a 50-step run, there are only 80 passes through the model instead of 100, and quality generally remains high. ## 6.6 Video Generation Models ![Figure 6.10: Video generation models take text prompts and may take keyframes or other image, audio, and video input.](https://www.datocms-assets.com/104802/1788123514-inference-engineering-figure-6-10.png) _Figure 6.10: Video generation models take text prompts and may take keyframes or other image, audio, and video input._ Video generation is the most demanding modality. Whenever possible, these models should be run on Blackwell GPUs (or Rubin, once available). These GPUs offer a high memory capacity for Context Parallelism, fast Tensor Cores for attention computation, and microscaling data formats for more precise quantization. Architecturally, video generation is similar to image generation, just rendering a full video rather than a single frame from latent space. Following the principle that greater scale unlocks more techniques, video generation uses all the same techniques as image generation plus additional optimizations. Like image generation, video generation is compute bound and works via iterative denoising over latent space. Video generation models generally take about the same number of denoising steps as image generation models (~50), but each step processes much more data. As video generation models are compute bound, batching isn’t useful like it is for text generation. Video generation models usually run on full nodes of eight GPUs with a batch size of one: all eight GPUs work together to create one video at a time. Unlike with batched workloads, where latency-throughput tradeoffs are possible by adjusting the batch size, the only way to improve the throughput and cost of video generation is to make the model itself faster. Early video generation models were framewise. They generated frames one at a time. This reduced the quality and coherence of the video output. Today, video generation models run denoising steps on the video as a whole in latent space. Where latent space for image generation represents two dimensions (width, height), for video models it represents three (width, height, time). This means passing huge amounts of data through each attention calculation. For video models, attention is 70 to 80 percent of the compute time, making attention the most important thing to optimize. ### 6.6.1 Attention Optimization and Quantization Attention optimization starts with kernel selection. Test FlashAttention, DeepGemm, CuTe, and CUTLASS kernels to see which ones perform best for your model. Where language models use the KV cache to accelerate attention, video generation models use other caching patterns to attempt to reuse model outputs. Re-using parts of the attention computation can make video generation 30 to 40 percent faster in practice. Precise methods and algorithms are continuously changing with new research, but there are two fundamental approaches to caching: - **Timestep-based caching:** Caching and re-using the outputs of certain timesteps to skip entire steps. - **Transformer-based caching:** Caching and re-using hidden states to skip layers within the transformer itself. Algorithms and implementations range from negligible quality degradation to unusable output – test these strategies carefully before using them in production. Beyond kernels and caching, the main tool for speeding up attention is quantization. For bandwidth-constrained language model inference, the benefit of quantization is that it means you have less data to load through memory. For video models, it means you access double the FLOPS by switching to lower-precision Tensor Cores. However, language model quantization focuses on weights – large linear layers where the impact of quantization is negligible. For video models, quantizing weights still helps, but while these layers take the majority of the memory bandwidth, constraining language models, they’re only a small fraction of the compute time for video models. Instead, quantization on video models focuses on attention. Attention is the riskiest part of any model to quantize, as errors accumulate over the course of inference. For video models, where there are ~50 steps instead of the thousands of autoregressive iterations in token generation, the risk is slightly lower but still important. The first method for reducing the quality impact is to use a blockwise quantization and a microscaling data format (MXFP8), both available on Hopper and Blackwell. Microscaling data formats do a better job of preserving outlier values, which have a major impact on attention accuracy. The most sophisticated approach to attention quantization is selectively quantizing within the model by: - **Step:** Keep early steps in FP16 and quantize later steps. - **Layer:** Keep first and last layers and quantize hidden layers. Quantization by step follows the same insight as the classifier-free guidance trick from image generation models: early steps establish the outline of the image, while later steps refine the details. These early steps are more important for prompt adherence and accuracy. For layers, the first and last layers are more important as they take the input and produce the final output. The hidden layers only perform intermediate calculations which don’t suffer as much from approximation. By only quantizing less important parts of the video generation process, quality is preserved. These tactics are found in kernels like SageAttention, an 8-bit attention kernel that you can use for quality low-precision attention on video generation models. ### 6.6.2 Context Parallelism While video generation models generally run on a full node of eight GPUs, they use Context Parallelism rather than Tensor Parallelism. Context Parallelism copies the weights onto every GPU. Video models are small enough that replicating the weights eight times takes a meaningful amount of memory but is feasible on B200. Instead of splitting the model across GPUs, Context Parallelism works by splitting the attention calculation across the GPUs. This is coordinated via a mechanism like ring attention, where each GPU holds a piece of the context and passes intermediate results to the next GPU in the ring. ![Figure 6.11: Context Parallelism replicates model weights but shares latent space to compute attention during video model inference.](https://www.datocms-assets.com/104802/1788123522-inference-engineering-figure-6-11.png) _Figure 6.11: Context Parallelism replicates model weights but shares latent space to compute attention during video model inference._ Attention for transformer models is multi-head, usually with eight or more heads. Attention heads are independent, so they can be run separately with the results combined afterward. Attention isn’t the only thing that can be parallelized. For example, the latent decoding step using the variational autoencoder takes three to five percent of the total inference time and can be run across GPUs. These parallelism techniques make AI video feasible. As video sequences get longer and video models get larger, parallelism will continue to be the most critical technique for video generation model inference.