2.2 LLM Inference Mechanics

LLMs are autoregressive token generation models. An LLM generates new tokens one at a time based on every previous token.

These tokens, the atomic units of language models, are numbers that represent chunks of text. Modern LLMs use subword tokenization, meaning that each token is a word or a fraction of a word.

Converting text into tokens and tokens into text does not require any neural networks. Instead, a tokenizer is a simple mapping between strings and their numerical token representation.

A language model’s vocabulary is the complete mapping between tokens and strings. Vocabularies and tokenizers vary from model to model, with more recent models employing more efficient tokenization schemes; the fewer tokens required to generate an output, the faster the end-to-end inference.

Most models have over 100,000 tokens in their vocabulary. In other modalities, like speech synthesis, the model vocabulary is expanded to let tokens represent other information like audio waveforms.

Inference involves two or three sequences of tokens:

• Input sequence: The prompt, chat, context, functions, and other input passed into the LLM.
• Reasoning sequence: Optionally, for reasoning models, an intermediate output sequence for thinking.
• Output sequence: The response generated by the LLM.

Combined, these sequences are limited to the model’s context window (the total number of tokens the model can process and generate per request). A request may further limit the output sequence length with a max_tokens argument.

While the input sequence is a single string, LLMs are trained to accept varied inputs like multi-turn chat sequences with roles, function signatures for tool calls, and, in some cases, multimodal inputs. These inputs need to be combined into a single sequence. This is handled by the chat template, which differs subtly from model to model and must be implemented correctly in the inference engine.

Tokenizing the input sequence with the chat template applied is step zero for inference. Then, there are two primary phases of inference:

• Prefill: Process the input sequence to calculate attention for each input token and store those values in a KV cache.
• Decode: Perform forward passes through the model to generate tokens autoregressively.

Each forward pass in the decode phase must generate a token. This takes a few extra steps, as neural networks output vectors, not tokens.

The output layer of the neural network used in LLM decode generates a vector of logits. The length of this vector equals the model’s vocabulary size. After normalization, these logits represent the probability of each potential token in the vocabulary being the correct output.

The output token is selected via a weighted random number generation based on the normalized probability vector. You can nudge that process via inference arguments:

• Temperature: Adjust the logits themselves before normalization.
• Top-k: Select the k most likely tokens after normalization, then re-normalize among them.
• Top-p: Select the smallest set of tokens after normalization whose probabilities add up to p.

A lower temperature, top-k, or top-p makes LLM output more predictable as the model is constrained to selecting highly likely tokens. Setting the temperature to 0 or top-k to 1 makes token selection deterministic (always select the highest-probability token).

When generating structured output, where the output conforms to a schema like JSON, there are additional tools like logit biasing for further directing the output of an LLM. These apply after each forward pass and support important LLM abilities like tool use; their correct implementation is essential for high-quality inference.

This logit generation and token selection process continues until the model decides that the stop token, a special value signifying the end of the output sequence, is generated (unless the context window or max tokens limit is hit first).

The two key pieces of this inference loop are generating the KV cache during prefill and generating output tokens (or more specifically the logit vectors that become tokens) during decode. These steps take the overwhelming majority of the time and resources during inference as they rely on large neural networks.

2.2.1 LLM Architecture

Every LLM on Hugging Face (the largest repository of open models) includes a config.json file: a few dozen lines detailing the model’s architecture.

The architecture of a model is a collection of decisions made during the training process about the nature and shape of each component of the model. Within a single architecture, there may be:

• Multiple sizes: Models at different parameter counts, like Llama 8B and 70B.
• Multiple variants: The “base” and “instruct” variants of a given model share the same architecture.
• Unlimited fine-tunes: Methods like LoRA (Low-Rank Adaptation) change behavior, not architecture.

These architectures matter because they determine runtime and engine support. If you have a highly optimized deployment of a given architecture, you can deploy another variant of the same architecture and enjoy the same performance improvements.

Model architecture is one of the first lines in most configuration files. To parse an architecture name like Qwen3MoeForCausalLM:

• Qwen: The model family, or the brand name of the model.
• 3: The major version of the architecture within the family.
• MoE: Indicates a Mixture of Experts model (see 2.2.4).
• CausalLM: Indicates a causal language model.

A causal language model predicts the next token in a sequence based on previous tokens, as opposed to, for example, a masked language model which fills in the blank based on surrounding tokens to the left and right. All generative LLMs today are causal language models.

Beyond the architecture’s name, the config.json file contains information about the nature and dimensions of the various layers that form the underlying neural networks of a model and the vectors that pass through them during inference.

2.2.2 Transformer Blocks

The main body of an LLM is a series of dozens to hundreds of transformer blocks. These blocks form the core of a large neural network with three kinds of layers:

• Embedding layer: The input layer of the neural network takes tokens and returns embeddings.
• Transformer blocks: The hidden layers within the network are transformer blocks that generate a prediction.
• Output layer: Also known as a language modeling head or LMHead, converts the hidden states from the transformer blocks into a vector of logits, one for each token in the model’s vocabulary.

Within the transformer blocks, there are sublayers for attention, a feed-forward neural network, and normalization.

The feed-forward neural network is a multi-layer perceptron. These linear sublayers make up the majority of the trainable weights within an LLM, while the attention sublayers are the second-largest component. Other components like normalization and activation functions are a rounding error in the model’s size.

While linear sublayers are the largest portion of the weights, the more complex operation for inference is attention.

2.2.3 Attention

Attention is the mechanism transformers use to relate a given token to other tokens in the sequence. Humans are good at interpreting the relationship between words. Attention brings the same capability to LLMs.

Consider the sentence “I decided to write a book because I thought it would be easy, but it was actually hard.” Attention shows that the word “it” in the sentence refers to writing a book.

The standard form of attention is scaled dot-product attention, as shown in this equation.

Attention takes three inputs:

• Q (queries): The embedded representation of the token being generated or updated.
• K (keys): Representations of all prior tokens.
• V (values): Computed attention values for all prior tokens.

Attention sublayers within models are multi-head, where each head is one attention operation. If you visualize the architecture of a neural network, these heads are parallel to each other on the same sublayer. Each head could be responsible for attending to different kinds of relationships between tokens, like subject-verb agreement and co-reference resolution.

There are two main types of attention:

• Self-attention: Q, K, and V are all from the same sequence.
• Cross-attention: Q is from a different sequence than K and V, conditioning Q on external information.

LLMs use self-attention (with a causal mask to prevent looking ahead in the sequence), while image generation and multimodal models also use cross-attention (e.g., between the image being generated and the associated text prompt).

Because attention looks for relationships between the current token and every previous token in the sequence, it’s a quadratic-time equation with respect to sequence length. As context expands, attention slows.

In practice, attention is linear, not quadratic, thanks to the KV cache. The KV cache is a standard component in attention implementation and stores key-value pairs for each previous token. By looking up this information in the KV cache rather than recomputing it each time, attention runs in linear time.

The KV cache is built during LLM prefill, used and updated during decode, and lives on GPU memory by default. Storing, accessing, and re-using the KV cache, a major topic in inference engineering, is covered in detail in section 5.3.

2.2.4 Mixture of Experts Models

The density of a neural network is determined by the number of connections between layers. Denser networks retain more information, while sparser networks take less compute and memory to run.

Mixture of Experts (MoE) is an architecture optimization that adds sparsity to linear layers. Rather than a single giant matrix, an MoE model has hundreds of smaller matrices (the experts) and routes each input to a small selection of experts. This is called activating the experts.

For an MoE model like Qwen3-235B-A22B, 22 billion parameters out of the model’s 235 billion total parameters are activated per request. Thanks to their low number of active parameters, MoE models are highly efficient for single-request local inference. However, in batched inference on production servers, different requests activate different experts, and you should expect almost all of the model parameters to be active at any given time unless sparsity is achieved in large-scale Expert Parallelism (section 5.4.2).

Expert routing is granular. Each forward pass through a model generates one token by working through every layer of the model. The router, a tiny model within the LLM, picks which experts to activate at each layer of the model. In the Qwen example, with 128 experts, the router picks eight experts at each of the 94 layers for every token that is generated.

MoE architectures are especially popular for larger models with 100B+ parameters, though there are MoE models as small as 20 to 30 billion parameters. Mixture of Experts unlocks a new form of inference parallelism called Expert Parallelism, which enables high-throughput inference for large models on multiple GPUs.

Models under 32B parameters, and especially models under 8B parameters, tend to use traditional dense architectures efficiently. Domain-specific models for tasks like tab completion also don’t gain much benefit from MoE as the entire model is effectively one expert.