The modality of a model describes what types of input it accepts and what types of output it creates. Chapters 1 through 5 focus on inference engineering for LLMs, which take text as input and produce text as output. This chapter expands the discussion to more modalities.
Generative AI models offer a rich array of modalities, including:
| Input | Output | Category |
|---|---|---|
| Text and image/video | Text | Vision language |
| Text or image/video | Vector | Embedding |
| Audio (voice) | Text | Transcription |
| Text | Audio (voice) | Speech synthesis |
| Text | Audio (music) | Music generation |
| Audio (voice) | Audio (voice) | Speech-to-speech |
| Text and/or image | 3D model | Generative CAD |
| Text and/or image | Image/Video | Image/video generation |
| Image/video | Text | Captioning |
| Image/video | Mask | Segmentation |
| Text and image | Image | Image editing |
Fortunately, while there are many modalities, there are just two broad archetypes of generative AI models as outlined in chapter 2:
- Autoregressive token generation: Start from a tokenized sequence and predict the most likely next token.
- Iterative denoising: Start from random noise and refine toward the most likely output.
LLMs are the most famous autoregressive transformers models for token generation, but far from the only ones. Vision language models, text and multimedia embedding models, automatic speech recognition (ASR) models, text-to-speech (TTS) models, and many others rely on similar architectures.
Many of the same inference engines and techniques used for LLMs also apply to these related modalities.
Image and video generation models instead rely on iterative denoising, though increasingly hybrid diffusion transformer models are setting the frontier in quality. While a number of the same philosophies from kernel selection to parameter tuning also apply to image model optimization, the details end up quite different.
For each new modality, you also need to adjust the way you think about and measure latency, throughput, and quality. For example, a single token of audio output from a TTS model isn’t particularly useful; instead of TTFT, measure the time to first word or time to first sentence.
This chapter discusses inference engineering for six common modalities beyond LLMs, with special attention to the different considerations for each modality.
- 6.1 Vision Language Models: Serving models that accept images and video alongside text, and the encoder costs that come with them.
- 6.2 Embedding Models: High-throughput embedding inference for search, RAG, and semantic similarity workloads.
- 6.3 ASR Models: Automatic speech recognition for transcription and dictation, including streaming audio and Whisper-family serving.
- 6.4 TTS Models: Speech synthesis under the latency requirements of real-time voice agents.
- 6.5 Image Generation Models: How serving image generation differs from LLM serving, and the optimizations specific to it.
- 6.6 Video Generation Models: The most computationally demanding modality, and what it takes to serve it in production.