There are two ways that you can add AI models to your product:
- Shared inference: Send your traffic to a public API endpoint for a given model and pay per million tokens or some other consumption-based metric.
- Dedicated deployments: Rent GPUs and set up an inference service exclusively for your application, paying per hour of GPU time (or purchase and install GPUs on-premises).
Shared versus dedicated inference is not exactly the same conversation as closed versus open models – there are plenty of shared endpoints for open models and many model labs offer large customers some kind of dedicated setup for their closed models. However, one of the key motivations for adopting open models is that it unlocks unrestricted dedicated inference.
Most AI products start with pay-per-token APIs because the tradeoffs make sense while looking for product-market fit or in the early stages of growth.
| Pros of shared inference | Cons of shared inference |
|---|---|
| Zero overhead, only pay for consumption | Cost scales linearly with usage |
| No cold start times, model is always available | Provider uptime caps product uptime, noisy neighbors |
| Minimal engineering work, just need an API key | No control over latency, model quality, or rate limits |
Over time, shifting AI products to dedicated deployments is essential for three reasons:
- Scale: You are processing enough volume of traffic that it’s more economical to pay per GPU than per million tokens.
- Specialization: You are running a custom or fine-tuned model, or you have specific latency or uptime requirements.
- Orchestration: Your product relies on multiple models and multi-step pipelines and you need to minimize network latency and deployment complexity.
The switch to dedicated deployments puts you in charge of your own inference engineering. This gives you flexibility and control but adds to your engineering surface area and increases the floor of your monthly spend on inference. Only switch once there is a clear and immediate business need.