Vision language models (VLMs) take one or more images or videos as input along with a text prompt and generate a text response.
A vision language model usually consists of two modules:
- LLM: A standard large language model.
- Vision encoder: A small model that takes raw images and videos as input and converts them into image tokens.
The language model is much larger than the vision encoder. For example, in Mistral Large 3, the vision encoder is just two billion parameters compared to the 673B-parameter LLM.
While the vision encoder is small by parameter count, it is critical for inference. VLMs use varied architectures and implementations for vision encoders, so runtime support is somewhat more fragmented for vision language models. This fragmentation increases the importance of vLLM and SGLang for serving vision language models.
As a rule of thumb, sending a high-resolution input image to a VLM adds about a thousand visual tokens to the input sequence. While at a very high level image tokens are similar to regular tokens, they add up quickly.
Across VLMs, the primary challenge in inference optimization is handling the longer input sequence and larger KV cache. This adds wrinkles at both phases of inference:
- Prefill: Images are patched, embedded, tokenized, and fed into prefill as part of the input sequence.
- Decode: Same mechanics, longer context, and some models add attention variants for the image tokens.
Every technique from the previous chapter is useful in addressing this challenge:
- Quantization: KV cache quantization reduces the memory bandwidth and storage overhead for longer sequences.
- Speculation: Decode for VLMs matches LLMs and can be accelerated with speculation, especially EAGLE.
- Prefix caching: Re-use KV cache for images in multi-turn chats and repeated queries.
- Parallelism: Use Tensor Parallelism for fast inference while accessing more VRAM for large models and long contexts.
- Disaggregation: Move prefill to specialized and independently scaling workers to handle long sequences.
In addition to these techniques, VLMs introduce a new quality-speed tradeoff: downsampling. Images and videos can be converted into visual tokens at various resolutions. A high-resolution representation takes about four times more tokens than a low-resolution image, but provides more detailed information. Downsampling generally isn’t needed for single-image inputs, but it may be needed when passing in multiple images or video clips.
6.1.1 Video Processing for Vision Language Models
A video is more than the sum of its frames. Videos may contain audio (though many VLMs cannot process audio, which must be transcribed separately and added into the prompt) and their frames express motion of objects through space that is lost when looking at static images.
VLMs are trained on video clips to understand that time dimension. High-quality inference requires processing the entire video clip in a single call to the model.
One second of cinematic video contains 24 frames. Each frame is an image. If a high-definition input image takes about 1,000 tokens to represent, then a four-second video clip produces an input sequence of nearly 100,000 tokens.
In reality, video inputs don’t generate quite this long of an input sequence – downsampling is practically obligatory.
Reducing the resolution and frame rate makes it possible to evaluate an entire clip in a single inference request, though video understanding models are still only capable of taking very short clips.
After the video is tokenized and encoded, inference is similar to working with images, just with much longer context. Prefix caching, KV cache offloading, and optimized attention implementations are of even greater importance for these input sequences of tens of thousands of tokens.
6.1.2 Omni-Modal Models
Vision language models are an important part of a trend toward “omni” models that accept multiple types of input and produce multiple types of output. There are pros and cons to omni models – their blend of modalities provides unique capabilities, but smaller specialized models are often faster and more accurate within specific domains.
For example, many VLMs have text recognition capabilities trained into their image input processing. However, these capabilities lag behind dedicated optical character recognition (OCR) models that are generally a fraction of the size.
Running production inference on VLMs often involves coordinating a pipeline of multiple models and pre-processing steps. You might have individual preprocessors for extracting data from PDFs, reading text from images via OCR, or transcribing audio from a video.
Each component in the pipeline must be individually optimized for speed and should scale independently to avoid bottlenecks.