# 1.2 About Your App

_Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 1: Prerequisites. [Full book index](https://www.baseten.co/inference-engineering/llms.txt)

Every inference engineering decision you make will be downstream of your use case.

Imagine a sports coach recruiting at a school. Which students would they want to talk to? It would depend entirely on which sport they’re coaching – the basketball coach would want the tallest kids in the class, but the gymnastics coach would select among the shortest.

Similarly, the way your inference system will be used determines how you go about building it.

There are two cases where inference engineers need to build at the highest level of generality:

- **Foundation models:** You have trained your own model and will sell consumption directly via a public shared inference API and need to support many usage patterns.
- **Inference platforms:** You are building an inference platform, either internally or as a product, and need to support any model and any use case.

But most AI-native applications are vertical apps like code editors or customer service agents, where AI is used to create some novel user experience. When building inference for vertical AI-native apps, you want to add as many constraints as possible by getting specific with your use case.

## 1.2.1 AI-Native Applications

Generative AI models have unlocked a new class of applications across industries and domains. Each category of application relies on different models and modalities and requires tailored inference.

| Category   | Example                                        | Considerations                                  |
| :--------- | :--------------------------------------------- | :---------------------------------------------- |
| Agents     | Prospecting agent for sales teams              | One user action triggers many inference calls   |
| Chat       | Front-line customer support chat with RAG      | Time to first token makes chat feel fast        |
| Voice      | Real-time translation between languages        | End-to-end latency for natural conversation     |
| Media      | Virtual try-on for clothes, shoes, and jewelry | Balance output quality vs. speed                |
| Search     | Legal document discovery                       | Offline corpus filling vs. online user requests |
| RecSys     | E-commerce product recommendations             | Consistent latency with high request volume     |
| Completion | Tab completion for coding in an IDE            | Full completion chunk at user’s typing speed    |
| Moderation | Scan user-generated content for safety         | High throughput for cost-effective checks       |

This is a tiny sample of the AI-native applications that are being built today, but it shows the breadth of considerations that inference engineers encounter. And as models get faster, cheaper, and smarter, new use cases that haven’t even been imagined today will emerge.

## 1.2.2 Online versus Offline

One of the primary tradeoffs in inference engineering is latency versus throughput. Lower latency makes your application faster, but higher throughput makes it cheaper at scale because you can use fewer GPUs for the same number of users.

Most AI applications – code completion, chat, voice agents – are online applications that run in real time. With an impatient user waiting on the other end of every API call, these online applications should be optimized for latency.

However, some applications have offline batch inference needs. Offline jobs are better served by high-throughput model deployments where each individual request would be too slow for a good user experience, but the system as a whole processes far more requests per hour in parallel.

Some example offline workloads include:

- **Catalog transcription:** Transcribing a back catalog of podcasts, interviews, or other audio to make it searchable.
- **Document processing:** Embedding, converting, or analyzing a set of documents on a regular cadence.
- **Corpus preparation:** Cleaning, embedding, or otherwise preparing massive corpora of data for model training.

It’s possible to have a single model that is used for both online and offline jobs. Whisper, a speech-to-text model, could be used in both a real-time dictation app and a batch transcription job. Assuming both use cases have enough volume, it will be more cost effective to create two separate deployments for the same model, with one optimized for latency and the other optimized for throughput.

## 1.2.3 Consumer versus B2B

Applications built for consumers and businesses have different inference needs.

Consumer applications are generally much more cost-sensitive and have less predictable usage patterns. Many consumer AI apps are designed for virality, and a single launch or marketing campaign can drive a spike in usage overnight.

Inference engineers working on consumer apps should prioritize marginal cost and flexibility while keeping latency and availability at a decent standard.

Business-to-business products often have better margins and more stable usage but require high availability and consistently low latency. Mission-critical software in the revenue path is held to a high standard of performance and reliability.

Inference engineers building for businesses must favor latency and uptime, though cost and scale are important secondary concerns.

In both consumer and business applications, compliance can limit infrastructure options, especially in regulated industries. Some essential considerations include:

- **Data sovereignty:** Are your GPUs in a geographic region where you are allowed to send user data?
- **User privacy:** Are the inputs and outputs of your model kept private and secure?
- **Regulatory compliance:** Are you and your underlying providers compliant with all relevant regulations?

Inference engineers must work closely with security and legal experts to ensure that the infrastructure they are operating is compliant.