# 2.3 Image Generation Inference Mechanics

_Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 2: Models. [Full book index](https://www.baseten.co/inference-engineering/llms.txt)

Image generation models take a text prompt and create an image based on the prompt. These models slightly predate the public rise of LLMs, with both closed models from Midjourney and open Stable Diffusion models first released in the summer of 2022.

Image generation models aren’t monolithic models like LLMs. Instead, they are pipelines of multiple models working together to generate images. Within a foundation model for image generation, there are three essential components:

- **Text encoder:** Converts the text prompt into instructions that the image generation model can understand.
- **Denoising model:** The heart of the model, iterates from noise to an image based on the prompt.
- **Variational Autoencoder (VAE):** Converts the model output from latent space to pixel space.

This pipeline mentality extends through the image model ecosystem. Beyond the base model, image inference often includes:

- **LoRAs:** Lightweight fine-tunes to change style and enhance quality.
- **ControlNets:** Outlines and edges to steer output images to match broad shapes and colors.

The vast and rich open-source ecosystem around image generation models includes tools like ComfyUI for building complex pipelines of foundation models and adaptions, swapping components to produce unique outputs.

The entire image generation pipeline operates in latent space. An ordinary HD image may be 1024x1024 pixels; that’s well over a million pixels. As the denoising model needs to calculate attention over the entire image in parallel, it would be infeasible to work in pixel space.

Latent space is a low-dimensional representation of an image. A latent space matrix for an image may be 128x128, or about one percent of the total values of the pixel space it represents.

The latent space is initialized as random values, or noise. The denoising model refines that noise into an image over a series of steps based on the text prompt. Each step updates the entire latent space, unlike LLMs which process tokens one at a time. Most image generation models take 30 to 50 steps to create a high-quality image.

![Figure 2.10: Diffusion-based models iteratively generate an image from noise, generally over 30 to 50 steps.](https://www.datocms-assets.com/104802/1788123234-inference-engineering-figure-2-10.png)
_Figure 2.10: Diffusion-based models iteratively generate an image from noise, generally over 30 to 50 steps._

Within each step, the model runs two forward passes: one with conditioning (the text prompt) and one without conditioning. These generations are then combined based on a guidance scale. Because of this two-part step, a 50-step image generation actually takes 100 forward passes.

This process, and other essential parts of image generation, are controlled on a request-by-request basis via inference arguments. The most important arguments are:

- **Prompt:** Describes what the image should look like.
- **Negative prompt:** Separately describes any styles or objects that should not be in the image.
- **Number of steps:** Trades off speed and quality with the number of denoising steps, 30 to 50 for most models.
- **Guidance scale:** Controls the balance between creativity and prompt adherence, integer value generally around 4.
- **Image size:** Selects from a fixed menu of resolutions and aspect ratios for the output image.

While these core mechanisms are common across image generation models, their architecture has evolved considerably in the past few years.

## 2.3.1 Image Generation Model Architecture

Image generation models are built on transformers, specifically diffusion transformers. A diffusion transformer is very similar to the transformers that LLMs use, but instead of processing embedding representations of discrete tokens, it processes image data.

Diffusion transformers look at images in patches. When training a text-to-image model, the images in the training data are fed in via overlapping patches of 2x2 or 4x4 pixels, which are then embedded into latent space. Inference works in the opposite direction, with latent space transformed back to pixels once the image generation is final.

Image generation models are pipelines of multiple models, including a text encoder, denoising model, and variational autoencoder. A clean example of this pipeline is Stable Diffusion XL (SDXL). SDXL is an old model, but its architecture remains relevant.

![Figure 2.11: SDXL architecture pipeline, adapted from “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis” (Podell et al., 2023).](https://www.datocms-assets.com/104802/1788123242-inference-engineering-figure-2-11.png)
_Figure 2.11: SDXL architecture pipeline, adapted from “SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis” (Podell et al., 2023)._

SDXL’s pipeline contains two diffusion models for denoising, the base and refiner. These models were trained for separate tasks: the base model goes from pure noise to a coherent image, while the refiner model adds details and ensures prompt adherence.

Modern models substantially outperform SDXL with better prompt adherence; accurate faces, hands, and details; legible text rendering; and support for image-to-image inference. These models, like the Qwen Image family, are broadly similar to the SDXL pipeline, but with larger and more capable models at every step.

| Component    | SDXL (2023)      | Qwen Image (2025) |
| :----------- | :--------------- | :---------------- |
| Text encoder | CLIP-based model | Qwen 2.5 VL (7B)  |
| Denoiser     | <4B parameters  | 20B parameters    |
| VAE          | Single encoding  | Dual encoding     |

The substantial growth in capability in modern image generation models like Qwen Image comes from larger component models and more complex pipelines. New abilities like legible text rendering and photorealistic human faces comes from switching from tiny NLP models to full LLMs for text encoding and increasing parameter counts on denoisers by a factor of five.

These larger models take more resources to run. Fortunately, as models have grown larger, GPUs have grown more powerful, though inference engineers can’t rely on hardware gains alone to run these models efficiently.

The latest research direction in image generation models is blending diffusion transformer architecture with LLM architecture.

Anything that can be tokenized can be modeled as an LLM. LLMs have the advantage of baked-in text understanding, and they are already an important component of image generation pipelines.

LLMs solve many of the problems inherent to diffusion models. Where diffusion models can only produce a fixed size of output, LLMs are autoregressive and can produce a variable-length output. And where image models need up to 100 forward passes to generate an image, LLMs generate tokens in a single forward pass. Models like HunyuanImage-3.0 use this LLM-style architecture.

While this is a new frontier in image generation, there are other existing architectures for accelerating image creation and for extending image models to video generation.

## 2.3.2 Few-Step Image Generation Models

The most time-consuming part of image generation is the 30 to 50 denoising steps. Rather than making each step faster, what if there was a way to optimize image models by simply using fewer steps?

Few-step image generation models are trained to do just that: create high-resolution images with eight or fewer denoising steps. These models are 80 to 90 percent faster out of the box than traditional image generation models, though their output quality is noticeably lower.

There are two primary methods for creating these models:

- **Latent consistency:** Train a model to predict the target latent image vector directly and repeat the prediction two to four times to enhance quality.
- **Distillation:** Use adversarial distillation and/or progressive distillation to train a small model to emulate a larger one in fewer inference steps.

Distillation is more common than latent consistency today. When new image models like FLUX and Qwen Image are released, members of the open image model community create distillations in addition to quality and style-oriented LoRAs.

If you have a latency-sensitive use case where quality is less important, like real-time generative filters, consider few-step image generation models.

## 2.3.3 Video Generation

Video generation models are architecturally similar to image generation models, just bigger. They have three to five times more parameters and encode ten to one hundred times more information in latent space.

The naive approach to video generation is to work frame-by-frame. Early video generation used this framewise approach: first generating a starting frame, then using that frame to generate the next frame, and so forth.

The issue with a framewise approach is error accumulation. Small issues early on compound with each frame, and the video goes off the rails quickly.

Instead, modern video models hold the entire video in latent space and modify it on each denoising step. Each frame attends to each other frame and is updated on every forward pass.

If latent space for an image model represents two physical dimensions, X and Y, then latent space for a video model represents three dimensions: X, Y, and T (time).

The main limitation of this approach is that videos are a fixed number of frames, just like image generation models have fixed aspect ratios. Modern video generation models create sequences of a few seconds.

The constraint on video length is compute resources. Even with the latest GPUs, attention over massive latent space is extremely expensive, taking several seconds of inference for each second of video. Video generation models are so compute-intensive that they typically run with a batch size of one, meaning a full node of eight GPUs is working on a single request.

While attention on each denoising step is expensive, video generation models have the same total number of steps as image models, generally around 50 steps.

Video generation is a recent modality compared to LLMs and image generation. Their limitations line up with the limitations of LLMs two years ago.

| LLMs (Late 2023)       | Video gen models (Late 2025) |
| :---------------------- | :---------------------------- |
| High TTFT, Low TPS      | Slow generation times         |
| Frequent hallucinations | Unrealistic physics           |
| Maxed out Ampere GPUs   | Max out Blackwell GPUs        |
| Limited context windows | Short video outputs           |

Today, these limitations are mostly eliminated from LLMs. Removing them from cinematic video generation, as well as related areas like world models, 3D object generation (XYZ dimensions rather than XYT), and other generative AI rendering models is a highly active area of research.

One important direction in research is coming back around to the idea of autoregressive generation, like with blending LLM architecture into image generation models. Rather than pure framewise video generation with its unusable error accumulation, new techniques like Self Forcing combine a global view of quality with an iterative approach to generation.

Adding autoregressive components to video models can partially address the bottleneck of attention, though it remains the most important and expensive component of inference.