5.3 Caching

During prefill, the inference engine builds a KV cache (a store of keys and values for each token) on the input sequence. It then updates the KV cache for each token during decode. As inference is autoregressive, the value for each new token depends on the value of every previous token in the sequence.

Every inference engine uses KV caching by default on a request-by-request basis. Without KV caching, LLM inference would be unbearably slow as each previous value in the entire sequence would need to be re-calculated for each subsequent token.

However, engineers can get even more utility from the KV cache by re-using it between requests rather than solely within each inference sequence.

5.3.1 Prefix Caching and KV Cache Re-Use

Consider the following two prompts, each with four tokens on most tokenizers, in Figure 5.7.

By default, the inference engine has to run prefill on all four tokens of each prompt. But the first tokens of each prompt – “Weather in” – form a shared prefix between the pair.

With prefix caching, you can re-use the KV cache from the first request to improve TTFT on the second request by skipping prefill on the first two tokens and reading in the existing KV cache instead.

When you see pay-per-token APIs charge less for “cache hit” input tokens than “cache miss” tokens, this is why – re-using cached tokens takes very little compute power or time. As an inference engineer, you can apply the same principle to reduce latency and improve throughput (thus saving money) on your own deployments.

Saving two tokens won’t make a big impact on TTFT, but prefix caching can skip prefill on thousands of tokens in certain domains:

• Complex system prompts: Agents, customer-facing chatbots, RAG scaffolds, and tool calls often feature long, complex system prompts on every call.
• Code completion: Code completion, code generation, and other coding functions require passing the same thousands of lines of code as shared context.
• Documents and retrieval: Document summarization, question answering, and retrieval all add repeated context ahead of user prompts.
• Multi-turn conversations: Ordinary conversations repeat back every message in a chat template, increasing the savings from prefix caching with every turn.

Prefix caching works from the start of the input sequence until the first non-repeated token. The fourth token in the weather example, a question mark, is shared between the two input sequences. However, the prefix ends at the first non-repeated token, so the fourth token isn’t read from cache.

Because prefixes end at the first unique token, your context engineering determines your TTFT savings. Consider a different approach to the same prompt:

Here, there is no savings from prefix caching as the very first token differs between the two sequences, even though every subsequent token is the same.

To take advantage of prefix caching, ensure that novel tokens are as late in your context as possible.

Prefix caching is the dominant form of KV cache re-use because LLMs are autoregressive. Each token influences every subsequent token, so a single novel token changes the way the model represents the rest of the sequence internally, even if the sequences look the same to a human reader.

However, there is active research around other kinds of KV cache re-use to overcome this limitation. Caching arbitrary sequences from the middle of prompts requires correcting both positional embeddings and selectively recomputing KV entries to maintain output quality. Tools like CacheBlend and LMCache support non-prefix sequences, expanding the possibilities for KV cache re-use.

5.3.2 Where to Store the KV Cache

The KV cache is very valuable. But KV caches take up a lot of memory, and GPUs only have limited VRAM.

You can configure how much memory your inference engine allocates to KV cache. For example, in TensorRT-LLM, you would set:

If you’re working on a B200 GPU with 180 GB of VRAM and used 100 GB for model weights and buffers, this would allocate 80 percent of the remaining VRAM, or 64 GB, to KV cache.

Once this allocation fills – and it will fill quickly – you’ll have to start deleting saved KV caches, increasing the chance of a cache miss on future requests.

To get more room for KV cache, offload from VRAM to other nearby storage. There are four places where you can store KV cache, in descending order of bandwidth to the GPU:

Certain SKUs, like the GB200, come equipped with CPUs and interconnects offering much faster G2 storage making them great for KV cache offloading.

NVIDIA Dynamo provides support for KV cache offloading via KVBM (KV Block Manager). KVBM provides APIs for moving KV cache blocks among different levels of memory. As a general rule, you want to keep the most frequently used blocks in higher-bandwidth memory, while less-often-used blocks can be relegated to slower storage until needed.

5.3.3 Cache-Aware Routing

In a production environment, there will be multiple replicas of your inference server, with incoming traffic split across the replicas. Usually, traffic is routed based on how busy each replica is.

If your inference server makes heavy use of prefix caching, your routing logic needs to be updated to account for that. A user in a long conversation with a chatbot or asking multiple questions about a codebase should have their request routed to the same replica whenever possible so that they get a cache hit for a faster, less expensive request.

Another option is using the G4 networked storage to build a global KV cache across replicas. Routing still matters here – a replica with a hot G1 cache will serve the request faster than a replica reading from G4 – but a global cache ensures that all replicas can eventually access any pre-computed sequence and that cached sequences are not lost when nodes cycle or are spun down during autoscaling.

5.3.4 Long Context Handling

“Long context” is a bit of a tautological definition: a sequence becomes “long context” when it generates a KV cache large enough to cause problems during inference.

Depending on the model, hardware, engine, and traffic, these problems can start to emerge past common cutoffs like 32K, 64K, or 128K tokens. In your performance benchmarking, be sure to send very large input sequences to test your inference service against long context requests.

Foundation model labs have been using scaling techniques like RoPE to unlock longer and more accurate context windows. But supporting these upgraded context windows introduces new challenges in inference.

Accounting for the KV cache, the attention equation scales linearly with sequence length. With long sequences, attention can become the main consumer of VRAM – the very resource decode is limited by.

While approaches like sliding window attention, compressed attention, and sparse attention offer solutions on a model-by-model basis, there are general approaches to optimizing the standard attention algorithm:

• FlashAttention: A series of optimized attention kernels to compute attention with reduced numbers of reads from and writes to memory.
• PagedAttention: A memory management technique that stores KV cache in fixed-size pages, reducing fragmentation and duplication.
• Chunked Prefill: A strategy of splitting large input sequences into chunks, which can be run alongside decode as resources allow to avoid overwhelming the inference engine with a long sequence.

But what if, after these optimizations, you still need more VRAM than a single GPU offers to store KV cache? You’ll need to parallelize inference across multiple GPUs.