# Chapter 1: Prerequisites _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) Inference engineering adds speed and scale to AI products by optimizing production serving of generative models. Optimization means identifying the best solution out of a range of options. Before optimizing model performance and building robust infrastructure, you need to know what “best” means for your product – many performance improvements come from making tradeoffs among latency, throughput, and quality. In practice, optimization is often about finding the right balance rather than maximizing a single factor. NFL players are big, fast, and strong. But they’re not as big as sumo wrestlers, as fast as Olympic sprinters, or as strong as champion powerlifters. Their bodies and skills are optimized to fulfill the specific demands of their position over the course of a full season. ![Figure 1.1: Just like elite athletes, inference services must be specialized for the demands of their workloads.](https://www.datocms-assets.com/104802/1788123090-inference-engineering-figure-1-1.png) _Figure 1.1: Just like elite athletes, inference services must be specialized for the demands of their workloads._ Similarly, your inference system must be optimized to fulfill the specific demands of your model, your product, and your traffic. The more constraints you can introduce, the better outcomes you can achieve. You should know your: - **Model requirements:** Which model(s) do you need to run inference on? - **Application interface:** How will inputs be delivered to the model, and how is the output expected to be formatted? - **Latency budget:** End-to-end, how fast does your product need to respond to a user action? - **Unit economics:** What does it make sense to spend on a per-request, per-user, or per-month basis? - **Usage patterns:** How many concurrent users are you serving, and is there any pattern to their usage (e.g., more activity during business hours)? Early on in building an AI product, the answers to these questions may not be clear. At this early stage, it’s often better to use off-the-shelf APIs whenever possible rather than investing in dedicated inference. As the product scales, requirements become clear and inference engineering becomes a worthwhile pursuit. ## 1.1 Scale and Specialization There are two ways that you can add AI models to your product: - **Shared inference:** Send your traffic to a public API endpoint for a given model and pay per million tokens or some other consumption-based metric. - **Dedicated deployments:** Rent GPUs and set up an inference service exclusively for your application, paying per hour of GPU time (or purchase and install GPUs on-premises). Shared versus dedicated inference is not exactly the same conversation as closed versus open models – there are plenty of shared endpoints for open models and many model labs offer large customers some kind of dedicated setup for their closed models. However, one of the key motivations for adopting open models is that it unlocks unrestricted dedicated inference. Most AI products start with pay-per-token APIs because the tradeoffs make sense while looking for product-market fit or in the early stages of growth. | Pros of shared inference | Cons of shared inference | | :--------------------------------------------- | :----------------------------------------------------- | | Zero overhead, only pay for consumption | Cost scales linearly with usage | | No cold start times, model is always available | Provider uptime caps product uptime, noisy neighbors | | Minimal engineering work, just need an API key | No control over latency, model quality, or rate limits | Over time, shifting AI products to dedicated deployments is essential for three reasons: - **Scale:** You are processing enough volume of traffic that it’s more economical to pay per GPU than per million tokens. - **Specialization:** You are running a custom or fine-tuned model, or you have specific latency or uptime requirements. - **Orchestration:** Your product relies on multiple models and multi-step pipelines and you need to minimize network latency and deployment complexity. The switch to dedicated deployments puts you in charge of your own inference engineering. This gives you flexibility and control but adds to your engineering surface area and increases the floor of your monthly spend on inference. Only switch once there is a clear and immediate business need. ## 1.2 About Your App Every inference engineering decision you make will be downstream of your use case. Imagine a sports coach recruiting at a school. Which students would they want to talk to? It would depend entirely on which sport they’re coaching – the basketball coach would want the tallest kids in the class, but the gymnastics coach would select among the shortest. Similarly, the way your inference system will be used determines how you go about building it. There are two cases where inference engineers need to build at the highest level of generality: - **Foundation models:** You have trained your own model and will sell consumption directly via a public shared inference API and need to support many usage patterns. - **Inference platforms:** You are building an inference platform, either internally or as a product, and need to support any model and any use case. But most AI-native applications are vertical apps like code editors or customer service agents, where AI is used to create some novel user experience. When building inference for vertical AI-native apps, you want to add as many constraints as possible by getting specific with your use case. ### 1.2.1 AI-Native Applications Generative AI models have unlocked a new class of applications across industries and domains. Each category of application relies on different models and modalities and requires tailored inference. | Category | Example | Considerations | | :--------- | :--------------------------------------------- | :---------------------------------------------- | | Agents | Prospecting agent for sales teams | One user action triggers many inference calls | | Chat | Front-line customer support chat with RAG | Time to first token makes chat feel fast | | Voice | Real-time translation between languages | End-to-end latency for natural conversation | | Media | Virtual try-on for clothes, shoes, and jewelry | Balance output quality vs. speed | | Search | Legal document discovery | Offline corpus filling vs. online user requests | | RecSys | E-commerce product recommendations | Consistent latency with high request volume | | Completion | Tab completion for coding in an IDE | Full completion chunk at user’s typing speed | | Moderation | Scan user-generated content for safety | High throughput for cost-effective checks | This is a tiny sample of the AI-native applications that are being built today, but it shows the breadth of considerations that inference engineers encounter. And as models get faster, cheaper, and smarter, new use cases that haven’t even been imagined today will emerge. ### 1.2.2 Online versus Offline One of the primary tradeoffs in inference engineering is latency versus throughput. Lower latency makes your application faster, but higher throughput makes it cheaper at scale because you can use fewer GPUs for the same number of users. Most AI applications – code completion, chat, voice agents – are online applications that run in real time. With an impatient user waiting on the other end of every API call, these online applications should be optimized for latency. However, some applications have offline batch inference needs. Offline jobs are better served by high-throughput model deployments where each individual request would be too slow for a good user experience, but the system as a whole processes far more requests per hour in parallel. Some example offline workloads include: - **Catalog transcription:** Transcribing a back catalog of podcasts, interviews, or other audio to make it searchable. - **Document processing:** Embedding, converting, or analyzing a set of documents on a regular cadence. - **Corpus preparation:** Cleaning, embedding, or otherwise preparing massive corpora of data for model training. It’s possible to have a single model that is used for both online and offline jobs. Whisper, a speech-to-text model, could be used in both a real-time dictation app and a batch transcription job. Assuming both use cases have enough volume, it will be more cost effective to create two separate deployments for the same model, with one optimized for latency and the other optimized for throughput. ### 1.2.3 Consumer versus B2B Applications built for consumers and businesses have different inference needs. Consumer applications are generally much more cost-sensitive and have less predictable usage patterns. Many consumer AI apps are designed for virality, and a single launch or marketing campaign can drive a spike in usage overnight. Inference engineers working on consumer apps should prioritize marginal cost and flexibility while keeping latency and availability at a decent standard. Business-to-business products often have better margins and more stable usage but require high availability and consistently low latency. Mission-critical software in the revenue path is held to a high standard of performance and reliability. Inference engineers building for businesses must favor latency and uptime, though cost and scale are important secondary concerns. In both consumer and business applications, compliance can limit infrastructure options, especially in regulated industries. Some essential considerations include: - **Data sovereignty:** Are your GPUs in a geographic region where you are allowed to send user data? - **User privacy:** Are the inputs and outputs of your model kept private and secure? - **Regulatory compliance:** Are you and your underlying providers compliant with all relevant regulations? Inference engineers must work closely with security and legal experts to ensure that the infrastructure they are operating is compliant. ## 1.3 Model Selection All else being equal – hardware, runtime, optimizations, architecture – inference on a smaller model with fewer parameters will be faster and cheaper than inference on a larger model with more parameters. That’s why the most important decision in model performance optimization isn't the runtime engine or speculation algorithm, it’s which model you choose to work with in the first place. AI engineers iterating on early-stage products should just use prebuilt pay-per-token APIs for powerful frontier models like Kimi and DeepSeek (or even closed models like GPT and Gemini). Before product-market fit, it’s not worth spending time or money on doing your own inference. But when it’s time to scale, the opposite advice applies. Find – or create – the smallest, easiest-to-run model that’s smart enough to handle the task at hand. In many cases, this is still going to need to be a trillion-parameter frontier model. But it’s always worth checking if a smaller, cheaper, faster model can get the job done. Which model you pick also affects what inference optimizations are available to you. Inference engines vary in the depth of support for different model architectures. Stick with popular model architectures to ensure you’ll find robust support across the performance tooling landscape. ### 1.3.1 Model Evaluation Model evaluation, or evals, is the practice of systematically measuring model intelligence. High conviction model evaluation is a prerequisite for inference engineering. Evals help inference engineers: - **Spend time wisely:** Before investing in making a model fast, evals ensure the model is useful. - **Establish a baseline:** Some performance optimization techniques risk reducing model quality, requiring a baseline to compare against. Unlike standard intelligence benchmarks, which measure model capabilities against common tasks like MMLU or SWE-bench, evals are tailored to specific products, domains, and tasks. Intelligence benchmarks are useful for shortlisting models, but they have become saturated or even gamed. Goodhart's Law states that "when a measure becomes a target, it ceases to be a good measure," and this applies to frontier labs’ heavy incentive to show new world-record intelligence benchmarks with each model release. While there are better ways to gauge overall model intelligence, like Elo rating on head-to-head win rate versus other models, there is no substitute for directly measuring how a model performs for your application. A few tips for doing useful model evaluation work: - **Look at your data:** Check eval results against your intuition for the product and problem space. - **Be precise:** Have a clear idea of the hardest problems a model needs to solve and focus evaluation there. - **Use tools:** Don’t reinvent the wheel on one of the fundamental problems in AI engineering. Appendix B includes recommendations for tooling and further reading about evals. ### 1.3.2 Fine-Tuning for Domain-Specific Quality Fine-tuning is the practice of taking a pre-trained foundation model and adapting it to a specific use case by introducing new data. ![Figure 1.2: Fine-tuning a model changes the values of the model weights while keeping the same overall architecture.](https://www.datocms-assets.com/104802/1788123095-inference-engineering-figure-1-2.png) _Figure 1.2: Fine-tuning a model changes the values of the model weights while keeping the same overall architecture._ If you can fine-tune a small model to pass your evals, you set yourself up for an easier time hitting your latency and cost targets for inference. A great example of a domain where fine-tuning is effective is translating English into SQL, a language used to query databases. General-purpose coding models are good at writing SQL, but these models are hundreds of billions of parameters. SQL is a relatively constrained language, so for an application that only needs to generate SQL queries from natural language prompts, a tiny fine-tuned model of just a few billion parameters can reach equivalent performance on this specific task. Text-to-SQL is an extreme example – many domains won't support such a vast reduction in model size – that illustrates what’s possible with a cleanly scoped domain, a strong set of evaluation criteria, and high-quality labeled data to use for fine-tuning. ### 1.3.3 Distillation What if you could retain most of the intelligence of a large model at a fraction of the size? That’s the idea behind distillation. ![Figure 1.3: Distillation preserves the behavior of a large model in a smaller, more efficient model.](https://www.datocms-assets.com/104802/1788123145-inference-engineering-figure-1-3.png) _Figure 1.3: Distillation preserves the behavior of a large model in a smaller, more efficient model._ Distillation is the process of using a large “teacher” model to train a smaller “student” model to emulate the larger model’s behavior. Unlike fine-tuning on synthetic data, where the model is trained on input-output pairs, distillation shows the student model the teacher model’s actual probability distributions, not just its final answers. Where fine-tuning teaches a model to perform better in a specific domain, distillation teaches the model how to emulate the behavior – good and bad – of a larger model. Distillation sees substantially less real-world use than fine-tuning. When a frontier lab releases a family of models of different sizes, the smaller models are generally not distilled from the larger ones. Instead, the models are independently trained to prevent the biases of the large models from artificially limiting the smaller models. But if the lab only trains a large model, distillation can make that model more accessible. In January 2025, open model research lab DeepSeek released their then-flagship reasoning model DeepSeek-R1. As the model was so large (671B parameters), they also released distilled versions of the model on top of the most popular open model architectures at the time: Llama 3 and Qwen 2.5. These distilled models showed similar reasoning behavior to the main DeepSeek-R1 model, albeit with worse intelligence benchmark scores, but the distilled models were relatively small and could take advantage of existing performance work for the Llama and Qwen architectures. At publication, these DeepSeek-R1 distills are still among the most popular distilled models on Hugging Face, along with distills of models like Whisper for audio transcription and some image generation models. ## 1.4 Measuring Latency and Throughput The two most common performance metrics for LLMs are TTFT (time to first token) and TPS (tokens per second). For modality-specific metrics beyond LLMs, see chapter 6. | Time to first token (TTFT) | Tokens per second (TPS) | | :------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------ | | With streaming output, how long does it take for a user to see the first output token? | How many tokens each second does the user receive after the first token is generated? | | Based on compute-bound prefill | Based on bandwidth-bound decode | | Lower TTFT == better latency | Higher TPS == better latency | While TTFT is a clear term, TPS is less precise. TPS can be a latency metric (tokens per second per user) or a throughput metric (tokens per second for the entire inference service). Most people use TPS to mean a per-user latency metric. When needed, use more specific terms: - **Perceived TPS:** The observed tokens per second per user after the first token (latency). - **Total TPS:** The total number of tokens generated each second by the inference service (throughput). - **Inter-token latency (ITL):** The time between subsequent tokens. An ITL of 10 milliseconds equates to 100 tokens per second per user. ![Figure 1.4: TTFT is the time it takes to generate the first token, while TPS measures how quickly subsequent tokens are generated.](https://www.datocms-assets.com/104802/1788123153-inference-engineering-figure-1-4.png) _Figure 1.4: TTFT is the time it takes to generate the first token, while TPS measures how quickly subsequent tokens are generated._ TTFT and TPS are most commonly used for user-facing LLM systems like chatbots, where output is streamed to the user. For other requests, like a tool call for an agent, you instead measure latency as total response time as the tokens aren’t useful individually. ### 1.4.1 Latency Percentiles One important distinction when discussing and comparing metrics is what percentile you are measuring. The naive approach is to simply look at an average (mean) TTFT or TPS. However, this does not tell the whole story. LLM total response time is generally a right-skewed distribution, where most times concentrate around a mode, but outliers can take significantly longer. ![Figure 1.5: Mean latency is generally higher than P50 latency due to outliers in the right-skewed distribution of inference times.](https://www.datocms-assets.com/104802/1788123161-inference-engineering-figure-1-5.png) _Figure 1.5: Mean latency is generally higher than P50 latency due to outliers in the right-skewed distribution of inference times._ These outliers can dramatically affect user experience and trust in a product. It’s not good enough for most interactions to feel snappy if one in every ten takes several seconds. Instead, inference engineers measure latency in percentiles. | P50 | Median latency | 1 in every 2 requests is slower | | :-- | :---------------------- | :-------------------------------- | | P90 | 90th percentile latency | 1 in every 10 requests is slower | | P95 | 95th percentile latency | 1 in every 20 requests is slower | | P99 | 99th percentile latency | 1 in every 100 requests is slower | While driving down average latency matters, good performance work also focuses on reducing P90/P99 latencies for a more reliable user experience. ### 1.4.2 End-to-End Metrics The other important distinction in metrics is whether you’re measuring solely inference time – the on-GPU time required to generate tokens – or an end-to-end measurement that accounts for network latency and any queue time. Both inference-only and end-to-end metrics are valuable to know. Inference time tells you how effective your model performance work is, while end-to-end metrics reveal your users’ perception of how fast your application is. When inference time is fast but end-to-end time is slow, turn your attention to infrastructure rather than model performance optimization.