# 6.3 ASR Models _Inference Engineering_ by Philip Kiely. © 2026 Baseten Labs, Inc. All rights reserved. From Chapter 6: Modalities. [Full book index](https://www.baseten.co/inference-engineering/llms.txt) Automatic speech recognition (ASR) models take audio as input and produce text as output, powering transcription and dictation apps. The most popular open ASR model is Whisper, which was released by OpenAI. Whisper supports dozens of languages with accurate transcription. ![Figure 6.4: ASR models transcribe input audio into text.](https://www.datocms-assets.com/104802/1788123486-inference-engineering-figure-6-4.png) _Figure 6.4: ASR models transcribe input audio into text._ Whisper comes in various sizes, but the largest and highest-quality Whisper model is just 1.55B parameters. Whisper runs extremely fast on fractions of large GPUs like H100 via Multi-Instance GPUs (MIGs). While various other sizes, variants, distillations, and quantizations exist, in practice it’s possible to satisfy most latency budgets with the highest-quality models: Whisper 3 Large and Whisper 3 Turbo. Whisper is an encoder-decoder model: - **Encoder:** Takes a processed audio waveform (log-Mel spectrogram) as input and encodes it into audio features. - **Decoder:** Takes these encoded audio features and converts them into text tokens. The overwhelming majority of inference time is spent on the decoder, which is an autoregressive transformer model very similar in architecture to an LLM. Fortunately, there are excellent tools for optimizing the main bottleneck. The main tool for performance optimization on the decoder side is TensorRT-LLM. With TensorRT-LLM, you can get in-flight batching for the decoder and an optimized C++ runtime with highly efficient CUDA kernels. TensorRT-LLM works especially well with recent architectures like Hopper and Blackwell, making MIGs an even better option for ASR inference. ## 6.3.1 Single-Chunk Latency Optimization One use case for Whisper is real-time transcription, like in a dictation app or voice agent. For live Whisper, look at round-trip time for a single chunk of audio to be transcribed. A great target to aim for is 200 milliseconds, which is the average human reaction time. With Whisper running on an optimized TensorRT-LLM inference engine, there isn’t a lot of work to do on the runtime level to improve performance. Instead, most gains for real-time Whisper come from orchestration and infrastructure. The biggest upgrade in product experience for ASR is streaming, which is implemented at the API server layer rather than the model runtime layer. By establishing a WebSocket connection (section 7.5.3) and streaming audio continuously in and text continuously out, products can transcribe in real time rather than transcribing pre-recorded audio. At the ASR runtime layer, nothing changes. Instead, a streaming implementation for transcription uses a Voice Activity Detection (VAD) model to monitor the incoming stream and segment it into discrete chunks for the ASR model to process. Inference is run on the chunks as normal, and the text results are streamed back via the WebSocket. This setup can handle several concurrent streams, and it has the advantage of keeping transcription sequential. When each chunk is processed on the same GPU, you can use the output sequence of the previous chunk as the prefix for the next chunk, improving transcription quality. ## 6.3.2 Long File Latency Optimization One limitation of the Whisper model is that it can only support 30-second chunks. Transcribing long files, like hour-long podcasts, requires a different set of optimizations. Measure the performance of long file transcription with the confusingly named Real-Time Factor (RTF). If the world’s fastest typist could manually transcribe an hour of audio in 30 minutes, they would have an RTF of 2X. With a Whisper deployment optimized for long files, you can transcribe an hour of audio in less than four seconds, for an RTF of 1000X. Fast transcription for long files requires a multi-step pipeline. The first step is again a VAD model, this time running on its own dedicated hardware. The model is used to remove silence and chunk out meaningful audio segments rather than splitting by time intervals, which runs the risk of cutting words in half. Then, the chunks can be processed in parallel. Ideally, you use multiple GPUs (or multiple MIGs) to process more audio chunks at once. RTF improves roughly linearly with the number of GPUs used. Each GPU processes multiple chunks at once with in-flight batching for high utilization. Finally, the chunked transcripts are stitched back together by timestamp. ![Figure 6.5: A two-stage pipeline for long audio file transcription parallelizes chunk transcription to improve end-to-end request time.](https://www.datocms-assets.com/104802/1788123491-inference-engineering-figure-6-5.png) _Figure 6.5: A two-stage pipeline for long audio file transcription parallelizes chunk transcription to improve end-to-end request time._ Parallelizing chunk transcription removes the ability to use the previous sequence as a prefix for the next sequence. But there are other quality improvement techniques that more than make up for this. With ASR output, you can automatically detect hallucinations like repeated words and phrases by measuring the compression ratio and words per minute of the output. When a chunk has an issue, you can: 1. Re-run the chunk with a higher temperature. This is counterintuitive – higher temperatures generally produce more hallucinations – but the intention is to break cycles of repeated words and generate a different output. 2. Re-chunk the entire audio, or a segment of the audio, into smaller chunks and re-run the transcription. In practice, these techniques obviate the need for passing a previous sequence as a prefix, unlocking highly efficient and accurate parallel transcription for long files. ## 6.3.3 Diarization Diarization, or annotating a transcript with who is speaking when, is an adjacent problem to transcription. Diarization models categorize audio by voice feature, then segment and cluster across the file to timestamp changes in speaker. Diarization models are a completely different class of model. Where Whisper is an encoder-decoder transformers model, diarization systems like pyannote audio are pipelines of classic ML models. A diarization pipeline contains models for segmentation, embedding, and clustering. To optimize diarization, you have to run each model fast and orchestrate the whole pipeline efficiently. As diarization is an ML pipeline, you can use tools like PyTorch and pyannote along with optimizations like Torch compilation to improve its performance. In practice, even highly optimized implementations of diarization take at least twice as long to process an audio file versus transcription.