How to Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization

Published September 23, 2026

NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model for identifying who spoke when in conversations. It supports up to eight speakers, overlapping speech, chunked processing, and configurable streaming latency. In Voice Arena's initial Diarization-Bench results, it ranked first with a 14.72% diarization error rate (DER).

Why speaker diarization matters

A conversation contains two kinds of information: what was said and who said it. Automatic speech recognition (ASR) transcribes words, while speaker diarization identifies the time intervals associated with each speaker. Combining both produces a speaker-attributed transcript.

A transcript can contain accurate words but still be difficult to use if no sentence is assigned to a participant. Without speaker attribution, applications may not know who made a commitment, raised an objection, or interrupted. Search, summaries, action items, conversation analytics, and voice-agent memory can all lose useful context.

Nemotron 3 Diarization supports eight anonymous speaker channels. It can report that speaker_2 spoke during a particular interval, but it does not determine that this channel belongs to a specific person. Applications can map channel IDs to known identities using meeting metadata, user profiles, or active speaker verification systems.

Earlier NVIDIA models, including Streaming Sortformer, established a similar approach for four-speaker diarization. The diar_streaming_sortformer_4spk-v2.1 checkpoint is used as the baseline in the evaluations described below.

How Nemotron 3 Diarization works

One model for offline and streaming conversations

Diarization systems must detect speech, assign it to the correct speaker, and maintain that assignment throughout a conversation. Streaming inference makes this difficult because the model receives only a small audio chunk and limited context at a time. Without a memory mechanism, a speaker assigned to one output channel in one chunk could receive a different channel in the next.

Nemotron 3 Diarization follows the Sortformer approach of ordering speakers by their first appearance. The first new voice becomes the first speaker channel, the next new voice becomes the second, and so forth. This arrival-time ordering keeps generic speaker labels stable across chunks and avoids solving a new speaker permutation for every chunk.

The model was trained with public and licensed speech data, including multispeaker-annotated real-world conversations licensed from David AI. Additional David AI audio supplied large-scale simulated English and multilingual mixtures covering 21 languages. According to the reported training analysis, adding this data reduced compound DER by 0.77 absolute points, from 11.19% to 10.42%, at both offline-style and ultra-low-latency operating points.

From audio to speaker activity

Nemotron 3 Diarization accepts 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step. The features are stacked by a factor of eight, producing 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings (RoPE).

The model uses a Conv1D layer above the Transformer to upsample predictions to the input-feature resolution. Its default output is a [T, 8] floating-point tensor, where T is the number of time steps and the eight columns represent possible speaker channels. Each value indicates the probability that a speaker is active at a particular time. The default stride is 10 ms and can be configured to another multiple of 10 ms.

Overlapping speech is represented naturally. If two people speak simultaneously, two speaker channels can be active in the same frame. Postprocessing converts the probabilities into generic speaker labels with start and end timestamps.

During streaming inference, two memory mechanisms provide context:

Arrival-Order Speaker Cache (AOSC): retains information about speakers observed in earlier chunks and organizes it by arrival-ordered channel.
First-in, first-out (FIFO) queue: provides recent frame context before the current chunk.

The input buffer also includes right context, which is audio immediately following the current chunk. More right context can help interpret speaker transitions, while less right context reduces the waiting time before output. Together, the current chunk, right context, FIFO queue, and speaker cache allow one model to operate at several latency levels.

Chunked inference removes a fixed model-imposed maximum recording duration. Accuracy can still decline on unusually long recordings or audio affected by severe noise, reverberation, far-field capture, or domain shift.

Diarization and speaker-attributed ASR are different tasks

Standalone diarization returns speaker activity and timestamps, not spoken words. ASR returns text but does not necessarily preserve speaker attribution. A speaker-attributed transcription pipeline combines both outputs:

Diarization identifies speaker activity intervals.
ASR produces text, ideally with word timestamps.
The application aligns words with the appropriate speaker intervals.

The distinction matters when evaluating a system. Diarization errors include missed speech, false speech detections, incorrect speaker assignments, and boundary errors. ASR errors affect the recognized words. Both components, as well as the combined pipeline, should be evaluated on representative audio.

Balancing latency and accuracy

Nemotron 3 Diarization supports recommended input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers can produce results sooner, while additional context generally improves accuracy and throughput.

These figures describe audio buffered before inference. Computation, networking, ASR, and application processing add to total end-to-end latency. Although the model can technically use an 80 ms input buffer, 0.32 seconds is the lowest recommended configuration.

Benchmark results

In Voice Arena's initial Diarization-Bench results, Nemotron 3 Diarization ranked first among 12 systems and 17 total configurations evaluated across 139 English-language conversations totaling approximately 22 hours.

With overlapping speech scored, system-generated speech activity detection, and no boundary collar, the model achieved a 14.72% DER. The next-ranked system recorded 19.3%, representing an approximately 24% relative reduction. Nemotron 3 Diarization also ranked first with 100 ms and 250 ms collars and across both in-person and online recordings. These initial results may change as Voice Arena completes its Version 1 evaluation and paired statistical analysis.

The broader evaluation included 901 condition-specific recordings covering multilingual telephone speech, meetings, near-field and far-field microphones, multi-microphone capture, and difficult acoustic environments. Overlapping speech was scored in every evaluation.

DIHARD III, AliMeeting, AMI, and NOTSOFAR1 used a zero-second collar, meaning no boundary tolerance was excluded from scoring. CALLHOME-Part2 used a 0.25-second collar. Results were generated with the NeMo e2e_diarize_speech.py evaluation script.

Measuring diarization accuracy with DER

Diarization error rate combines three types of error:

Missed speech: a reference speaker was active, but the system detected no corresponding speech.
False alarm: the system marked a speaker as active when the reference contained no corresponding speech.
Speaker confusion: the system detected speech at the correct time but assigned it to the wrong speaker.

Evaluation settings can materially affect DER, so the scoring protocol is part of the result. The comparison with NVIDIA's previous four-speaker streaming Sortformer uses the diar_streaming_sortformer_4spk-v2.1 baseline and the final Nemotron 3 Diarization values rather than preview-model results.

An average 41.0% relative DER reduction at 1.04 seconds

At 1.04-second input-buffer latency, Nemotron 3 Diarization reduced DER on all eight listed evaluation conditions. Relative reductions ranged from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.

The unweighted mean of the eight per-dataset relative reductions was 41.0%. This is an average of relative improvements across evaluation conditions, not a pooled DER calculated by combining every recording into one score.

The improvement was also consistent across operating points. At the shared latencies of 30.4, 1.04, and 0.32 seconds, the final model recorded lower full-set DER than the previous baseline on every evaluated dataset.

Improvements at higher speaker counts

Support for eight speakers is intended for meetings and group conversations with more than four participants. The benchmark advantage widened in the higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1.

The two-speaker CALLHOME subset shows an important exception: the final model recorded 5.98% DER, compared with 5.68% for the previous baseline. Across the complete CALLHOME-Part2 evaluation, however, DER improved from 10.32% to 9.10%, with larger gains in higher-speaker-count subsets.

DIHARD III groups recordings with five through nine speakers into one result. Because nine speakers exceed Nemotron 3 Diarization's supported maximum of eight, that aggregate includes audio outside the model's specified speaker-count limit.

Accuracy and throughput

The model card reports real-time factor speedup (RTFx), calculated as total audio duration divided by total processing time. A higher RTFx indicates that more audio can be processed per unit of compute time.

At the 30.4-second configuration, Nemotron 3 Diarization reached 15,113× RTFx at batch size 32 with torch.compile(), compared with 2,619× for the previous baseline. DIHARD III DER decreased from 19.09% to 12.73%.

At the 1.04-second configuration, the model reached 865× RTFx compared with 136× for the baseline, while DER decreased from 19.60% to 13.18%.

These measurements represent batched throughput on the disclosed test system. They are not measurements of single-stream, end-to-end application latency. Complete pipelines should be benchmarked on target hardware, including data movement, diarization, ASR, and downstream processing.

Getting started with NVIDIA NeMo Speech

Install dependencies

After setting up Python 3.12 or later, Cython, and a recent PyTorch version, install the system packages and NVIDIA NeMo speech dependencies:

apt-get update && apt-get install -y libsndfile1 ffmpeg
uv pip install Cython packaging
uv pip install 'nemo-toolkit[asr]'

Run offline diarization

Use a 16 kHz mono recording containing two or more speakers. The following example loads the checkpoint and applies the recommended 30.4-second offline-style configuration:

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "nvidia/Nemotron-3-Diarization"
)
diar_model.eval()

## Configuration values are measured in 80 ms frames.
diar_model.sortformer_modules.spkcache_len = 264
diar_model.sortformer_modules.fifo_len = 40
diar_model.sortformer_modules.chunk_len = 340
diar_model.sortformer_modules.chunk_right_context = 40
diar_model.sortformer_modules.spkcache_update_period = 300
diar_model._check_streaming_parameters()

predicted_segments = diar_model.diarize(
    audio=["/path/to/conversation.wav"],
    batch_size=1,
)

for segment in predicted_segments[0]:
    print(segment)

The diarize() method returns speaker-marked segments in the form start_seconds end_seconds speaker_id. An illustrative output structure is:

0.400 2.100 speaker_0
1.800 3.250 speaker_1
3.600 4.700 speaker_0

In this example, both speakers are active between 1.800 and 2.100 seconds. A speaker can appear in multiple segments, and intervals do not need to be mutually exclusive. To obtain the underlying speaker-activity tensors as well, set include_tensor_outputs=True.

The API accepts an audio path, a list of paths, NumPy arrays, or a line-delimited JSON manifest. For NumPy arrays, pass the correct integer sample_rate. The default file-audio sample rate is 16 kHz.

Nemotron 3 Diarization supports 16 kHz, single-channel .wav, .flac, .opus, and .mp3 audio. It is designed for Linux systems with supported NVIDIA Ampere, Hopper, or Blackwell GPUs.

Select a latency and quality configuration

Streaming parameters are specified in 80 ms encoder frames. The recommended configurations are:

Configuration	Input-buffer latency	Speaker cache	FIFO	Chunk	Right context	Cache update period
Offline style	30.4 s	264	40	340	40	300
Low latency	1.04 s	264	264	9	4	222
Very low latency	0.64 s	264	264	6	2	222
Ultra-low latency	0.32 s	264	264	3	1	222

Input-buffer latency is calculated as:

(CHUNK_LEN + RIGHT_CONTEXT) × 80 ms

The value excludes model computation, network transport, ASR, and application processing. Although an input buffer as short as 80 ms is technically possible, 0.32 seconds is the lowest recommended configuration. Lower latency generally reduces both accuracy and throughput, so the operating point should be selected according to end-to-end requirements.

Use all five parameter values from the same configuration row, then call _check_streaming_parameters() before running diarization. The speaker cache retains earlier speaker context, FIFO controls recent history, chunk and right context determine input-buffer latency, and the cache update period controls how much FIFO context is used for a cache update.

Combine diarization with offline ASR

Run ASR and diarization on the same recording and time base. For example, Parakeet TDT 0.6B v3 can return word timestamps. The following example associates each word with the speaker active at its midpoint:

from nemo.collections.asr.models import ASRModel

asr_model = ASRModel.from_pretrained(
    model_name="nvidia/parakeet-tdt-0.6b-v3"
)
words = asr_model.transcribe(
    ["/path/to/conversation.wav"], timestamps=True
)[0].timestamp["word"]

turns = []
for segment in predicted_segments[0]:
    start, end, speaker = segment.split()
    turns.append((float(start), float(end), speaker))

def speaker_at(midpoint):
    active = sorted({
        speaker for start, end, speaker in turns if start <= midpoint < end
    })
    if len(active) == 1:
        return active[0]
    return "overlap/ambiguous" if active else "unassigned"

for word in words:
    midpoint = (word["start"] + word["end"]) / 2
    label = speaker_at(midpoint)
    print(f"{word['start']:.2f}-{word['end']:.2f} {label}: {word['word']}")

The midpoint rule is a simple alignment heuristic. It marks simultaneous speaker activity as ambiguous and leaves words outside detected speech unassigned. It does not separate overlapping voices or establish which active speaker produced a particular ASR word.

Production systems should evaluate word-boundary alignment, overlap handling, and errors from both models on representative audio before combining words into speaker turns.

Deployment considerations

The model supports a maximum of eight speakers. If a recording contains more, speech can be missed or assigned to the wrong channel. Noise, severe reverberation, far-field recording, domain shift, and long conversations can also increase missed speech, false alarms, boundary errors, or speaker confusion.

Applications should preserve uncertainty rather than treating every speaker assignment as infallible. The complete system should be evaluated on audio representative of its intended environment, especially before speaker attribution is used in regulated, safety-relevant, or consequential workflows.

Use of the model is governed by the OpenMDW License Agreement, version 1.1.

How to Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization