The decode phase of LLM inference is an autoregressive process in which tokens are generated one at a time. The bottleneck on decode is memory bandwidth, with compute sitting idle at low to moderate batch sizes as weights are read from memory.
Speculative decoding takes advantage of that spare compute to try to generate multiple tokens per forward pass through the target model. If an inference engine could generate two, three, or even more tokens for each round-trip of weights through memory, it would generate far more tokens per second. Speculative decoding only improves TPS/ITL, not TTFT.
There are multiple algorithms for speculative decoding. They share a common mechanism:
- The speculator generates one or more draft tokens.
- The target model, or the underlying model that you’re trying to accelerate, performs validation on these tokens to check if they match what the model would have generated.
- The target model accepts any valid draft tokens and generates an additional token itself, completing the forward pass.
This generates N+1 tokens per forward pass, or iteration through the decode loop, where N is the number of accepted draft tokens.
Generating draft tokens is not free, it takes both compute and memory. However, it is much faster for a target model to validate a draft token than to generate an original token. If you imagine a sudoku puzzle, solving it is hard, but checking if the solution is correct is very easy. For the target model, generating a token is like solving a sudoku, while validating a draft token is like checking a finished sudoku.
The performance uplift from any speculative decoding strategy depends on three factors:
- Draft token cost: The time it takes to generate a draft token.
- Draft sequence length: The number of draft tokens that are generated per forward pass.
- Token acceptance rate: The percentage of draft tokens that are accepted by the target model.
Token acceptance rate is high early in the draft sequence, but draft tokens get less reliable deeper in the sequence.
Aim for short, high-percentage sequences because generating and validating tokens, while inexpensive relative to generating tokens in the original model, still comes with meaningful overhead. Additionally, once a single draft token is rejected as wrong, all subsequent tokens in the sequence are also rejected.
Working with speculation is interesting because so many factors affect token acceptance rate. The big one is temperature – higher temperatures yield token distributions that are harder to predict, reducing the effectiveness of speculative decoding. But even factors as simple as subject matter can make a difference on acceptance rate if the draft model or additional head used for speculation is better versed in, say, math than history.
Another limitation on speculative decoding is that it’s most useful at low batch sizes where there are spare compute cycles. At higher batch sizes, speculative decoding must be dynamically disabled as compute is too saturated to afford verification.
Each speculation algorithm navigates these tradeoffs differently, and careful implementation of the right algorithm for the situation can lead to major improvements in TPS.
5.2.1 Draft-Target Speculative Decoding
The original method of speculative decoding uses two models:
- Draft model: An additional model that generates speculative draft tokens.
- Target model: The original model, which now verifies draft tokens in addition to performing ordinary decode.
The most important decision when configuring draft-target speculative decoding is which draft model to use. A good draft model has a high token acceptance rate while requiring minimal resources to run.
Draft models are most often smaller members of the same family as the target model, as they share tokenizers and behaviors. As a rule of thumb, the draft model should be at least ten times smaller by parameter count than the target model. Fine-tuning or distillation can improve the token acceptance rate of these small models by teaching them to behave more like the target model.
Draft-target speculative decoding is a good choice when you want something that is quick to set up out of the box without doing any training or fine-tuning.
However, other speculation algorithms generally offer better performance. Draft-target introduces the most overhead of any speculative decoding method. While the draft model is small, the inference engine must store the draft model’s weights, activations, and KV cache in memory, and dedicate compute cycles to draft model prefill. Additionally, the draft and target model must run in coordination so that they don’t compete for resources, though inference engines like TensorRT-LLM handle that model orchestration.
5.2.2 Medusa
Medusa was one of the first alternatives to draft-target speculation. Medusa addresses the complexity and overhead of operating a draft model to perform speculation by instead fine-tuning the target model to generate additional tokens per forward pass.
Fine-tuning a model for Medusa means grafting additional decoder heads onto the model. The ordinary architecture of an LLM contains a single decoder head, but Medusa adds an additional two to four heads that generate sequential draft tokens.
Like with draft-target speculation, draft tokens are validated on the next forward pass.
Medusa is still limited on draft token count and draft token acceptance rate, and it is not widely used in production today. However, Medusa inspired more popular techniques like EAGLE.
5.2.3 EAGLE
The main problem with using an off-the-shelf pretrained model as a draft model is that a model like Qwen 0.5B is designed to be a good standalone LLM on cheap hardware, not to speculate draft tokens on a B200. These draft models are inefficient to run and offer a relatively low acceptance rate.
EAGLE offers an alternative: a purpose-built draft model trained from scratch to generate sequences of up to eight draft tokens (two times more than Medusa) with a very high acceptance rate.
During inference, LLMs accumulate a lot of context about the predicted tokens in the form of hidden states between layers. Traditional draft models don’t have access to this information.
EAGLE is a draft model trained to accept hidden states as input and generate speculative tokens as output. Specifically, it is trained on a set of three hidden states: one from an early layer, one from a middle layer, and one from a late layer. EAGLE is often less than one billion parameters and scales well when given additional training data.
In practice, when using EAGLE with an inference engine like TensorRT-LLM, implementations tend to be straightforward, with post-training EAGLE creation and single-sequence speculation.
EAGLE can be attached to the same module (PyTorch class) as the target model, so each forward pass runs inference on both the target model and the EAGLE speculator. This unified pipeline solves the other problem with draft-target decoding, where multiple round trips to the CPU were needed to orchestrate the draft and target models.
EAGLE is the go-to speculation algorithm for general use among inference engineers with the knowledge and means to train EAGLE heads and is well-supported by inference engines. Like other speculation techniques, adopting EAGLE for improved latency requires reduced batch sizes, lowering throughput and increasing cost.
5.2.4 N-gram Speculation and Lookahead Decoding
N-gram speculation uses a different mechanism than other types of speculation. There is no draft model.
Instead, in parallel with generating the KV cache, the inference engine constructs an n-gram dictionary. The n-gram dictionary maps a single starting token to an observed sequence of N tokens (the n-gram).
This n-gram dictionary contains common sequences from the input text and is first constructed during prefill. During decode, the generated token is fed into the dictionary, and any available suffix is selected as draft tokens. In the next forward pass, the target model verifies these draft tokens as normal.
The advantage of n-gram speculation versus EAGLE is that the sequences can be much longer. While EAGLE can generate eight or so draft tokens with a decent acceptance rate, n-gram sequences can exceed ten tokens.
However, the acceptance rate for n-grams is only high when the contents of the model output are similar to the model input. N-gram speculation is mostly used for code completion and code revision, where syntax is predictable and output closely matches input. However, within this specific domain, it easily outperforms EAGLE.
A similar method to n-gram speculation is Lookahead Decoding, which generates n-grams during inference to fill the dictionary. Lookahead Decoding is more general than n-gram speculation as it doesn’t rely as much on highly repetitive context, but it requires extra compute to generate the n-grams.
Every speculative decoding algorithm aims to reduce the total number of forward passes needed to generate a complete output sequence, improving overall latency and specifically tokens per second per user during decode. N-gram speculation excels at code completion and similar tasks, while Lookahead Decoding generalizes in systems with excess compute.