# 6.2 Embedding Models

_Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 6: Modalities. [Full book index](https://www.baseten.co/inference-engineering/llms.txt)

An embedding model transforms a variable-length chunk of text – or another modality of input like an image – into a fixed-length vector representation that captures the semantic meaning of the input.

![Figure 6.2: Embedding models convert unstructured input data into vectors that encode semantic meaning.](https://www.datocms-assets.com/104802/1788123473-inference-engineering-figure-6-2.png)
_Figure 6.2: Embedding models convert unstructured input data into vectors that encode semantic meaning._

By encoding content into this shared semantic vector space, you can compare distance between items with simple math. Embedding models (along with vector databases) are used to build agent memory, RAG, search, and recommendation systems.

To support these use cases, embedding model inference workloads have two different traffic profiles:

1. **High-throughput backfills:** Bulk operations like indexing millions of documents, updating product catalogs, or even preparing data for LLM pre-training.
2. **Low-latency lookups:** Individual user-facing queries for search, retrieval, or recommendation, where every millisecond affects user experience.

Inference engineering for embedding models starts with clarifying which profile you need to serve. If you need to do both and have enough traffic to justify the cost, it’s better to build a separate system for each type of usage.

## 6.2.1 Embedding Model Architecture

There are tens of thousands of embedding models on Hugging Face, but they all use one of two transformers-based architectures:

- **BERT-style models:** Encoder-only neural networks, usually <1B parameters, originally built for masked token prediction.
- **LLM-based models:** Modern language models, generally <=8B parameters, repurposed to generate embeddings.

Today, LLM-based embedding models offer substantially greater capabilities, though BERT-style models are still used for simple latency-sensitive tasks like classification.

Embedding models introduce their own speed/quality tradeoff in embedding dimensionality, or the size of their output vectors. An embedding vector contains a few hundred to a few thousand values, with longer vectors encoding more information.

Modern embedding models use Matryoshka representations to unlock dynamic tradeoffs between embedding dimensionality and quality while retaining more information on shorter vectors. Dimensionality doesn’t materially affect inference time but does affect the storage, retrieval, and similarity computation time within a system.

In most cases, vectors from one embedding model cannot be meaningfully compared to vectors from another embedding model, even if they are the same length, as they encode inputs into different semantic spaces.

## 6.2.2 Embedding Model Inference

For embedding models with LLM backbones, like Qwen 3 Embed 8B, inference optimization shares common tools and techniques with other high-volume, low-latency deployments of smaller LLMs.

There are multiple runtimes for text embedding models: vLLM, SGLang, Infinity, TEI (Text Embedding Inference by Hugging Face). But the best performance comes from adapting TensorRT-LLM to run these models.

![Figure 6.3: A high-performance embedding inference pipeline adds parallel tokenization and batch management in front of an optimized inference engine.](https://www.datocms-assets.com/104802/1788123478-inference-engineering-figure-6-3.png)
_Figure 6.3: A high-performance embedding inference pipeline adds parallel tokenization and batch management in front of an optimized inference engine._

TensorRT-LLM brings an optimized XQA kernel for fast attention and kernel fusion techniques to reduce memory access overhead. For supported models, TensorRT-LLM is the most performant inference engine for both latency and throughput.

Further gains come from quantization. While smaller models are more likely to lose quality from quantization, FP8 quantization for embedding model weights offers improved performance with minimal quality loss.

The easiest way to check embedding model quality post-quantization is to run the same inputs through both the original and the quantized model, then check the cosine similarity of the output vectors. A cosine similarity of one hundred percent means the vectors are identical; you’ll want to see at least 99 percent similarity to have confidence in the quantization.

As embedding models process tokens in parallel, prefix caching and disaggregation aren’t relevant optimizations. And given these models’ small size, parallelism across multiple GPUs is not effective. Instead, high-traffic deployments should scale horizontally, with each GPU as its own replica.

In high-traffic deployments of embedding models, batching and queueing play an important role in performance. Embedding models offer much higher batch sizes than other models. A single request may batch dozens or hundreds of text inputs together in a list, and many requests can run in parallel on a single GPU as even the most demanding embedding models are relatively small and fast.

Whether you’re performing a large backfill or handling a surge in usage, traffic can exceed even the large batch sizes offered by embedding models. In these cases, a robust queuing system is essential infrastructure for supporting embedding model inference.