NVIDIA released Nemotron 3 Diarization, an open-weight, 100-million-parameter model for real-time speaker identification and attribution in multi-speaker audio. The model ranks first on Voice Arena's Diarization-Bench leaderboard with a 14.72% diarization error rate (DER) and supports up to eight speakers in live and recorded conversations, a doubling of the four-speaker limit in NVIDIA's previous Streaming Sortformer baseline.
The model accepts 16 kHz single-channel audio, converts it to Mel-spectrogram features with a 10 ms frame step, and feeds those features through a 31-layer Transformer encoder with rotary positional embeddings. The output is a tensor of shape [T, 8]—T time steps by eight possible speaker channels—where each value represents the probability that a speaker is active at that moment. During streaming inference, two memory mechanisms preserve speaker identity across chunks: an Arrival-Order Speaker Cache (AOSC) that retains information about speakers observed earlier, organized by their arrival-ordered channels, and a first-in-first-out (FIFO) queue that supplies recent frame context. This design allows one model to operate at multiple latency points without retraining.
The model supports four recommended input-buffer latencies: 30.4 seconds for offline-style processing, 1.04 seconds for low latency, 0.64 seconds for very low latency, and 0.32 seconds for ultra-low latency. These values measure only the audio buffered before inference; computation, networking, ASR, and application processing add to end-to-end latency. At the 30.4-second configuration on an NVIDIA RTX PRO 5000 with batch size 32 and torch.compile(), the model reaches 15,113× real-time factor (RTFx)—meaning it processes audio 15,113 times faster than real time—compared with 2,619× for the previous baseline, while reducing DIHARD III DER from 19.09% to 12.73%. At the 1.04-second configuration, it achieves 865× RTFx versus 136×, lowering DER from 19.60% to 13.18%.
Compared with the previous four-speaker Streaming Sortformer baseline at 1.04-second latency, Nemotron 3 Diarization reduces DER on all eight evaluated datasets, with relative improvements ranging from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean of per-dataset relative reductions is 41.0%. The advantage widens in higher-speaker-count conditions: on DIHARD III recordings with five through nine speakers, the improvement is larger than on two-speaker subsets, though the model's eight-speaker maximum means recordings with nine speakers exceed its specified limit.
The model outputs generic speaker labels with timestamps, not identity verification. Downstream applications must map anonymous channel IDs to explicit speaker identities by pairing timestamps with meeting metadata, user profiles, or active speaker verification. Diarization errors include missed speech, false alarms, incorrect speaker assignments, and boundary errors. The Hugging Face blog post notes that noise, severe reverberation, far-field recording, domain shift, and long conversations can increase these errors. The model is designed for Linux systems with supported NVIDIA Ampere, Hopper, or Blackwell GPUs and accepts 16 kHz mono audio in .wav, .flac, .opus, and .mp3 formats.
For teams building voice-based agent systems or audio processing pipelines, the key trade is between latency and accuracy: shorter input buffers reduce response time but degrade both accuracy and throughput, so operating-point selection should follow end-to-end product requirements rather than buffer duration alone.