NVIDIA Nemotron 3 Diarization Tracks Up to Eight Speakers in Real Time

NVIDIA has introduced Nemotron 3 Diarization, an open-weight model with 100 million parameters that determines who spoke when in a conversation. In the first results of Voice Arena's Diarization-Bench it ranked first among 12 systems, evaluated on 139 English-language conversations totalling about 22 hours of audio. Its diarization error rate (DER) is 14.72%, against 19.3% for the next-ranked system — a relative reduction of roughly 24%. The description was published on the Hugging Face blog on 23 September.

Speech recognition captures what was said; diarization classifies who said it, marking the intervals during which each speaker is active — including moments when people talk over one another. Those timestamps can be combined with ASR output to build a speaker-attributed transcript.

How the model works

Nemotron 3 Diarization accepts 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step, stacked by a factor of eight into 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings (RoPE). The output is a [T, 8] tensor of speaker-activity probabilities: eight channels over T time steps. Two channels can be active in the same frame when speech overlaps.

During streaming, two memory mechanisms keep labels stable: the Arrival-Order Speaker Cache (AOSC) retains context on speakers seen in earlier chunks, and a FIFO queue supplies recent frames. Speakers are ordered by arrival time, so the first new voice becomes the first channel. Adding licensed David AI audio to training cut compound DER by 0.77 points, from 11.19% to 10.42%, NVIDIA reports.

Benchmarks and latency

The model supports recommended input-buffer latencies of 30.4, 1.04, 0.64 and 0.32 seconds; 0.32 seconds is the lowest recommended setting. At 1.04-second latency, DER improved on all eight evaluated conditions, with relative reductions from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM; the unweighted mean of the eight reductions is 41.0%. One exception: on the two-speaker CALLHOME subset the new model records 5.98% against 5.68% for the previous baseline.

Throughput rose as well: at 30.4 seconds the model reaches 15,113× real-time factor (RTFx) at batch size 32 with torch.compile(), against 2,619× for the baseline, and DIHARD III DER falls from 19.09% to 12.73%. At 1.04 seconds the figures are 865× and 136×, with DER dropping from 19.60% to 13.18%.

Limits and availability

The model supports a maximum of eight speakers; more participants can mean lost speech or misassigned channels, and noise or far-field capture also raise error rates. NVIDIA advises testing the full diarization-plus-ASR pipeline on representative audio before using speaker attribution in consequential workflows. The model is released under the OpenMDW License Agreement, version 1.1, runs on NVIDIA Ampere, Hopper or Blackwell GPUs, and appears in a Hugging Face Spaces demo.