The two most common performance metrics for LLMs are TTFT (time to first token) and TPS (tokens per second). For modality-specific metrics beyond LLMs, see chapter 6.
| Time to first token (TTFT) | Tokens per second (TPS) |
|---|---|
| With streaming output, how long does it take for a user to see the first output token? | How many tokens each second does the user receive after the first token is generated? |
| Based on compute-bound prefill | Based on bandwidth-bound decode |
| Lower TTFT == better latency | Higher TPS == better latency |
While TTFT is a clear term, TPS is less precise. TPS can be a latency metric (tokens per second per user) or a throughput metric (tokens per second for the entire inference service).
Most people use TPS to mean a per-user latency metric. When needed, use more specific terms:
- Perceived TPS: The observed tokens per second per user after the first token (latency).
- Total TPS: The total number of tokens generated each second by the inference service (throughput).
- Inter-token latency (ITL): The time between subsequent tokens. An ITL of 10 milliseconds equates to 100 tokens per second per user.
TTFT and TPS are most commonly used for user-facing LLM systems like chatbots, where output is streamed to the user. For other requests, like a tool call for an agent, you instead measure latency as total response time as the tokens aren’t useful individually.
1.4.1 Latency Percentiles
One important distinction when discussing and comparing metrics is what percentile you are measuring.
The naive approach is to simply look at an average (mean) TTFT or TPS. However, this does not tell the whole story. LLM total response time is generally a right-skewed distribution, where most times concentrate around a mode, but outliers can take significantly longer.
These outliers can dramatically affect user experience and trust in a product. It’s not good enough for most interactions to feel snappy if one in every ten takes several seconds.
Instead, inference engineers measure latency in percentiles.
| P50 | Median latency | 1 in every 2 requests is slower |
|---|---|---|
| P90 | 90th percentile latency | 1 in every 10 requests is slower |
| P95 | 95th percentile latency | 1 in every 20 requests is slower |
| P99 | 99th percentile latency | 1 in every 100 requests is slower |
While driving down average latency matters, good performance work also focuses on reducing P90/P99 latencies for a more reliable user experience.
1.4.2 End-to-End Metrics
The other important distinction in metrics is whether you’re measuring solely inference time – the on-GPU time required to generate tokens – or an end-to-end measurement that accounts for network latency and any queue time.
Both inference-only and end-to-end metrics are valuable to know. Inference time tells you how effective your model performance work is, while end-to-end metrics reveal your users’ perception of how fast your application is. When inference time is fast but end-to-end time is slow, turn your attention to infrastructure rather than model performance optimization.