Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable – without sacrificing the quality that makes them so valuable. Both improving performance and preserving quality require a strong intuition for how models work under the hood.
Generative AI models are a composition of big, complex neural networks.
The history of neural networks stretches back to the 1950s, when the first perceptrons for simple binary classification were implemented in hardware. In the following decades, perceptrons were abandoned but then reinvented from single to multi-layer perceptrons with a new concept, back-propagation, which introduced hidden states between layers and a learning procedure that repeatedly adjusts weights within the network.
These neural networks had only a few layers. In the 2000s, research began into deep neural networks with dozens of layers. In 2012, AlexNet became the first deep neural network to show promising real-world capabilities and the effectiveness of GPUs for deep learning, leading to new architectures like word embedding models for text and Generative Adversarial Networks (GANs) for images.
But the story truly starts in 2017, when Vaswani and colleagues published the seminal paper “Attention Is All You Need,” introducing the transformer. A transformer is a neural network with an attention mechanism that can learn relationships between various parts of a sequence.
Transformers are the foundation of generative AI. Transformers aren’t just for LLMs, they power every modality of model from embedding to voice to image and video generation.
Across modalities, there are two important styles of transformer-based models:
- Autoregressive token generation: Start from a tokenized sequence and predict the most likely next token.
- Iterative denoising: Start from random noise and refine toward the most likely output via diffusion.
This chapter explores the architectural details of LLMs (autoregressive token generation) and image generation models (iterative denoising).
- 2.1 Neural Networks: Linear layers, matrix multiplication, and activation functions as the compute primitives behind generative models.
- 2.2 LLM Inference Mechanics: Autoregressive generation, prefill and decode, transformer blocks, attention, and mixture of experts architectures.
- 2.3 Image Generation Inference Mechanics: Diffusion pipelines, few-step image models, and how video generation extends the same mechanics.
- 2.4 Calculating Inference Bottlenecks: Using ops:byte ratio and arithmetic intensity to determine whether a workload is compute bound or memory bound.
- 2.5 Optimizing Attention: Why attention scales quadratically with sequence length, and the algorithms that make long context tractable.