NVIDIA Nemotron 3.5 ASR is in the Baseten Model Library, enabling developers to deploy low-latency, production-ready speech recognition with just a few clicks. It ships as two 600M-parameter streaming models: a dedicated English model and a multilingual model supporting 40 language locales. In our testing, a single H100 sustained up to 100 concurrent real-time streams over WebSocket, with even higher throughput achievable over gRPC.
Two models, one streaming architecture
Nemotron 3.5 ASR is built to deliver both low latency and high transcription accuracy. It comes in two variants that share one cache-aware FastConformer-RNNT architecture (a 24-layer encoder and an RNNT decoder): a dedicated English model, and a multilingual model that handles 40 language locales from a single model. Language-ID prompt conditioning lets the model auto-detect the spoken language or use one you specify, and it produces punctuation and capitalization by default.
The models are released under the OpenMDW-1.1 license and are deployed on Baseten using NVIDIA Inference Microservices (NIM), providing highly optimized streaming inference with production-ready performance. We continue working with the NVIDIA team to distribute NIM releases as they become available.
Nemotron 3.5 ASR performance on Baseten
We benchmarked Nemotron on a single H100, serving NVIDIA’s Realtime WebSocket API with audio paced at real time:
Finalization latency (the time between a speaker finishing and the final transcription being ready) remained nearly flat as concurrency increased from 8 to 100 real-time streams (98 ms to 138 ms). This consistency is especially important for interactive AI agents, captioning, and voice applications. We also observed a stable time-to-first-token, which is determined primarily by the model’s 1-second lookahead window used to enhance transcription accuracy.A key performance advantage from NVIDIA NIM is cache-aware inference, which reuses encoder context across chunks instead of reprocessing overlapping audio buffers. This significantly improves throughput, enabling up to 6x more concurrent streams than NVIDIA’s Parakeet RNNT while maintaining low latency. While the prior measurements use a WebSocket, similar to other popular speech-to-text (STT) offerings, the NIM also exposes a gRPC interface, which carries less per-message serialization overhead than the WebSocket JSON path and can sustain considerably higher concurrency per replica for backends that can consume it.
One streaming model across 40 language locales
The multilingual model delivers strong accuracy across transcription-ready languages. On the FLEURS benchmark, it achieves an average 8.84% WER across the 19 supported languages, including 4.11% WER for Spanish, 4.25% for Italian, and 7.91% for English. For English-only workloads, the dedicated English variant provides the highest possible accuracy, reaching 2.32% WER on LibriSpeech Clean.
Our multilingual testing, based on FLEURS and a realistic, noisy telephony benchmark (μ-bench) aligns with findings of low WER across transcription-ready locales. The value is consistent real-time accuracy across a range of languages that few streaming models cover with only one model.
For specialized domains, under-resourced languages, or unique accents, developers can further improve transcription quality by fine-tuning Nemotron ASR using NVIDIA NeMo through Baseten Training. If you’re interested in fine-tuning the model for your specific use case, you can learn more in this blog.
Try Nemotron ASR Streaming
The English and multilingual variants of Nemotron ASR Streaming are now live in the Baseten Model Library. Deploy production-ready, low-latency speech recognition in a couple of clicks, or talk to one of our engineers to discuss your voice AI workloads.
