Inference Engineering

Chapter 1: Prerequisites

The framing decisions that come before optimization: app requirements, model selection, and performance measurement.

Inference engineering adds speed and scale to AI products by optimizing production serving of generative models.

Optimization means identifying the best solution out of a range of options. Before optimizing model performance and building robust infrastructure, you need to know what “best” means for your product – many performance improvements come from making tradeoffs among latency, throughput, and quality.

In practice, optimization is often about finding the right balance rather than maximizing a single factor.

NFL players are big, fast, and strong. But they’re not as big as sumo wrestlers, as fast as Olympic sprinters, or as strong as champion powerlifters. Their bodies and skills are optimized to fulfill the specific demands of their position over the course of a full season.

Figure 1.1: Just like elite athletes, inference services must be specialized for the demands of their workloads.
Figure 1.1: Just like elite athletes, inference services must be specialized for the demands of their workloads.

Similarly, your inference system must be optimized to fulfill the specific demands of your model, your product, and your traffic. The more constraints you can introduce, the better outcomes you can achieve.

You should know your:

  • Model requirements: Which model(s) do you need to run inference on?
  • Application interface: How will inputs be delivered to the model, and how is the output expected to be formatted?
  • Latency budget: End-to-end, how fast does your product need to respond to a user action?
  • Unit economics: What does it make sense to spend on a per-request, per-user, or per-month basis?
  • Usage patterns: How many concurrent users are you serving, and is there any pattern to their usage (e.g., more activity during business hours)?

Early on in building an AI product, the answers to these questions may not be clear. At this early stage, it’s often better to use off-the-shelf APIs whenever possible rather than investing in dedicated inference. As the product scales, requirements become clear and inference engineering becomes a worthwhile pursuit.

  • 1.1 Scale and Specialization: The two paths to adding AI to a product, shared model APIs versus dedicated deployments, and when to switch.
  • 1.2 About Your App: How modality, latency budget, unit economics, and traffic patterns shape every downstream inference decision.
  • 1.3 Model Selection: Picking a model through evaluation, then using fine-tuning or distillation to hit quality targets with something smaller.
  • 1.4 Measuring Latency and Throughput: TTFT, tokens per second, latency percentiles, and end-to-end metrics: defining performance before optimizing it.