Text-to-speech (TTS) models, also called speech synthesis models, take text as input and produce audio as output, specifically generating speech. In 2025, open models like Orpheus TTS introduced extremely lifelike speech synthesis to the open model ecosystem. Many companies fine-tuned Orpheus for increased vocal quality and product-specific voices, leading to high adoption of open models in the voice AI space.
Modern TTS models are fine-tuned LLMs. Orpheus TTS, for example, is derived from Llama 3.2 3B. This means that many of the same runtime and performance optimizations developed for LLMs apply to speech synthesis models.
TTS models have a small parameter count – Orpheus TTS at three billion is on the larger end – meaning that like ASR models, MIGs on H100s are highly efficient and performant options for inference.
Unlike ASR models, which generally run in FP16, TTS model weights and KV cache can be quantized to FP8 for better performance in addition to the optimized kernels and in-flight batching introduced by the TensorRT-LLM inference engine.
TTS models with LLM backbones are trained by expanding the vocabulary size of the LLM with tens of thousands of encoded audio tokens. Then, the models are trained on pairs of text inputs with tokenized audio outputs. This means that to use TTS models in practice, you also need an audio decoder that takes the audio output tokens and converts them into a waveform.
This audio decoding process adds a potential bottleneck to inference. The audio decoder should be implemented using PyTorch and compiled for efficient operation on the target GPU and should use dynamic batching with a short timeout (e.g., 15 milliseconds). In-flight batching is not possible for the audio decoder.
TTS model performance is measured with somewhat different metrics than LLMs. The key metrics are:
- TTFB: Time to first byte (TTFB) is the equivalent of TTFT for speech synthesis.
- Time to first sentence: Instead of TTFB, a more user-oriented latency metric is the time to generate the first meaningful phrase or sentence.
- TPS: Like an LLM, the TTS model generates tokens, so decode speed can be measured in tokens per second.
Like with TTFT on LLMs, the goal is to minimize TTFB for speech synthesis. For Orpheus, it’s possible to get as low as 150 milliseconds on a single H100.
However, there are different goals for TPS on speech synthesis models. The tokens that the model generates are converted to audio waveforms. Depending on the model, it might take 80 to 100 tokens per second to generate audio in real time. Beyond that level, there isn’t any benefit to generating additional tokens per second.
Instead, performance enhancements are used to scale throughput in terms of the number of concurrent real-time outputs the model can create. If a single GPU can support many concurrent users, the per-user cost of speech synthesis drops dramatically.
6.4.1 Streaming Real-Time Text to Speech
Most TTS tasks call for real-time speech synthesis. Like with ASR models, the performance gains for real-time systems come less from the runtime layer – which has already been optimized with TensorRT-LLM, quantization, and a compiled SNAC decoder – and instead from infrastructure.
Again, streaming over WebSockets is the biggest unlock for performance versus sending text and receiving audio in discrete chunks. After testing the inference engine to determine how many concurrent real-time streams can be generated, set the same batch size and active WebSocket count to keep usage high but stable.
TTS models are rarely used outside of real-time applications. However, if you do end up with a batch use case like backfilling a large corpus of documents to audio for improved accessibility, note that TTS models don’t do well with long inputs, speech starts to degrade after 30 seconds or so.
6.4.2 Speech-to-Speech Models
One exciting area of research is speech-to-speech models, or models that take audio as input and generate audio as output.
Today, most voice systems use a cascading approach, where an ASR model, LLM, and TTS model work in a pipeline to listen, think, and respond to users. These pipelines also employ auxiliary components like VAD and embedding models to facilitate natural conversation and add context.
Speech-to-speech models, like OpenAI’s gpt-realtime, augment a core LLM with audio consumption and production capabilities, effectively unifying the pipeline in a single model. This is possible thanks to ASR, LLM, and TTS sharing such similar architectures, especially on the decoder.
At the time of publication, there are no commercially viable open speech-to-speech models, and closed options like gpt-realtime are significantly less capable and more expensive than cascading multi-model setups. However, research in this space is robust, and this emerging modality will soon require its own flavor of inference engineering.