NVIDIA Nemotron 3.5 Transcribe and Granary v2: Promptable, Adaptable Multilingual Speech Recognition

NVIDIA Nemotron 3.5 Transcribe and Granary v2

NVIDIA Nemotron 3.5 Transcribe is an open-weight speech model for multilingual transcription, speaker attribution, translation, and audio question answering. It is accompanied by Granary v2, an open multilingual data-curation pipeline, and NeMo Speech fine-tuning tools for adapting recognition to specialized vocabulary, accents, and transcript formats.

A transcript can record every spoken word while still leaving important questions unanswered: who spoke, how the content should be translated, and how domain-specific terms should be written. Calls, meetings, classroom recordings, and medical dictation may each require different output formats, including speaker labels, translations, or normalized terminology.

Nemotron 3.5 Transcribe is available on Hugging Face and ModelScope. The model:

• Transcribes 40 languages.
• Translates 31 languages to or from English.
• Attributes speakers in 13 languages.
• Processes recordings of up to one hour.
• Labels up to eight speakers.
• Supports natural-language instructions and word-level timestamps.

The same checkpoint can be prompted to produce different results, such as:

• “Transcribe this recording and label each speaker.”
• “Translate this audio into English.”
• “What decisions were made in this meeting?”

NVIDIA reports that the model ranks first among open-weight models on the English Hugging Face Open ASR Leaderboard, with an average word error rate (WER) of 5.33%. It also reaches 2.8% AA-WER on Artificial Analysis, where approximately 4 billion of its 30 billion parameters are active per token. On David AI’s 21-language diarization leaderboard and VoiceArena’s 14-system comparison, it records the lowest reported speaker-attribution error rates.

Quantized versions are planned for local deployment on NVIDIA RTX Spark, DGX Spark, and Jetson AGX Thor systems, as well as on Blackwell data-center GPUs. Transcribe is designed for recorded audio rather than live streaming.

Prompting transcription and audio analysis

The model can follow instructions that specify the desired task, language, script, and transcript style. For unfamiliar terminology, a prompt can provide possible terms while requiring the model to use them only when supported by the recording. For example:

Transcribe this medical discussion in the original language. Possible terms: HbA1c, metformin. Use these terms only when supported by the audio. Preserve standard medical spelling and write numbers in conventional written form.

The model card provides recommended presets, including reasoning prompts for audio questions, a complete prompt guide, and a task-by-language matrix. Language support varies by task, and audio question answering is trained in English. Names, speaker changes, and answers should still be checked against the original recording.

Agora used word-level timestamps in a classroom demonstration. The system displays a recorded class transcript alongside the replay, allowing students to see both what was said and when it was said.

Benchmark results

Hugging Face Open ASR Leaderboard

The Hugging Face leaderboard evaluates transcription accuracy using WER on public test sets. Nemotron 3.5 Transcribe is reported as the leading open-weight model in three tracks:

1. English: 5.33% average WER.
2. Multilingual: 4.62% mean WER across seven languages.
3. Long-form: 10.98% mean WER.

Lower WER indicates fewer transcription errors. In the English comparison, the model also records an RTFx of 462.09.

Artificial Analysis

Artificial Analysis reports a tie for first among open-weight models at 2.8% AA-WER. Nemotron 3.5 Transcribe uses approximately 4 billion active parameters, compared with 24 billion for the other model in the tie.

On AA-AgentTalk, a held-out voice-agent speech set contributing 50% of the AA-WER index, Transcribe records 1.8% WER. The next-best open-weight model records 2.3%.

The model combines a speech encoder with a hybrid Mamba-Transformer mixture-of-experts language model. Although the model contains 30 billion parameters, approximately 4 billion are active for each token. This places it on Artificial Analysis’s open-weight accuracy-versus-active-parameter frontier.

Speaker attribution

Sortformer speaker cues are passed to the language model together with acoustic features. This allows Transcribe to identify speakers during generation rather than relying only on a later process to attach speaker labels to words.

On David AI’s DAI-ASR-I18N combined transcription-and-speaker-attribution track, Transcribe ranks first among evaluated open-weight models and second overall. On the diarization track, which measures who spoke when across 21 languages, it records an 18.26% diarization error rate (DER), the lowest among the 14 tested systems. NVIDIA’s Nemotron 3 Diarization follows at 18.38%.

On VoiceArena’s English full-session benchmark, Transcribe records the lowest concatenated minimum-permutation WER (cpWER) among 14 systems and ranks first in both in-person and online settings. Its cpWER is 32.4%, followed by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. The benchmark covers overlapping speech and sessions with two to eight speakers.

Transcribe’s deletion rate is 8.32% of reference words, compared with a median of approximately 31% across the evaluated systems.

Ocular AI’s Converse-STT evaluation ranks Transcribe as the leading open model. Josh Talks’ Voice of India leaderboard reports results across Indian languages.

Throughput and deployment

In benchmarking, the forthcoming NVFP4-quantized version processed 100 hours of audio in under three minutes on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. The test used 30-second clips at high concurrency.

The available checkpoint can be served with vLLM. Local deployment with NeMo-Speech.cpp is planned alongside the quantized versions. SGLang-Omni also provides deployment support.

Granary v2: An open speech-data pipeline

Granary v2 is the data-curation pipeline used to prepare multilingual speech for Nemotron 3.5 Transcribe. It is part of the Nemotron speech collection on Hugging Face and ModelScope, and is built with NVIDIA NeMo Curator.

The public manifest covers 463,062 hours across 48 languages and 16 source corpora, compared with 25 languages in Granary v1. These figures describe the released dataset and do not represent the complete training mixture or its sampling weights.

Granary v2 creates multiple training targets for each recording, including three transcript forms:

The pipeline also produces:

• Short audio captions.
• Restored code-switching and loanword forms.
• Speech-grounded question-and-answer pairs.
• Contextual ASR biasing terms.
• Translations with automatic quality scores.
• Provenance and per-record quality metadata.

Granary v2 has four reusable stages:

1. Audio Preparation and Metadata: Segments raw or long-form audio, tracks speakers, and identifies acoustic events and language. The released Granary v2 sources were already segmented, so this stage was skipped.
2. Transcription and ASR: Routes each language to primary and recovery ASR models. A fallback is used when the primary model fails a hallucination check. Human reference transcripts help retain trusted spellings while adding spoken forms and disfluencies.
3. Task Label Generation: Creates transcript variants and checks character rate, hallucination patterns, and agreement with model or human references. Accepted records receive contextual ASR terms and translations.
4. High Fidelity: Removes noisy, clipped, or multi-speaker segments for applications that require higher-quality audio. For VoxPopuli, this reduced 354,000 hours to 111,000 hours across 22 European languages.

Developers can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends on their own recordings or authorized public corpora. The public manifests include tn_raw, itn_raw, and itn_clean transcripts, task labels, entity hints, derived instruction pairs, provenance, and quality flags. Source audio must be downloaded separately under each corpus’s original license.

Adapting recognition to specialized vocabulary

Prompting is the first step for domain adaptation. In the medical case study described below, a prompt that specified English and the Latin alphabet reduced full-benchmark WER from 30.8% to 23.8%.

For errors that remain, NeMo Speech fine-tuning pairs representative speech with explicit written targets. The medical ASR case study uses Eka Care’s benchmark of recordings from Indian doctors and medical students. The benchmark contains 3,619 clips and has no designated training split.

The adaptation data combines:

• Approximately 153 hours of synthetic speech.
• 29 hours of accented English from Common Voice.
• 63 hours of medical recordings from MultiMed.

On the full test set, WER falls from 30.8% with the original model to 23.8% with the improved prompt and 11.8% after fine-tuning.

Synthetic data separates the written target from its spoken rendering. For example, HbA1c can be used as the written label while “H B A one C” is sent to the speech synthesizer. Multiple text-to-speech voices add variation, while real recordings contribute accents and recording conditions.

Training examples vary conventions such as digits, unit spacing, company-name expansion, and brand-name hyphenation. Instructions specify the desired convention. Some prompts include near-miss terms so the model learns to use a spelling only when the audio supports it. The final system uses one fixed convention prompt and one model pass, without retrieval or transcript post-processing.

The recipe trains the speech encoder and projection while using LoRA to adapt the language model. Two runs use different data mixtures. After adapter merging, selected checkpoints are uniformly averaged into one final model. Checkpoints are selected using decoded development-set results rather than training loss alone.

The complete adaptation process, from text generation through evaluation, took approximately 14 GPU-hours on one NVIDIA GB300.

Results on held-out speakers

The medical benchmark is divided by speaker into 855 tuning clips and 2,764 held-out clips. The tuning clips are used to select the prompt and checkpoints, while the held-out clips are decoded once for each final system.

Using Eka Care’s KARMA scoring pipeline, fine-tuning reduces held-out WER from 22.44% for the best-prompt baseline to 12.17%. This is a 45.8% relative reduction. Keyword WER falls from 9.2% to 6.5%, and semantic WER falls from 10.0% to 7.6%.

On the same held-out speakers and scoring pipeline, the adapted model records 12.17% WER, compared with 13.79% for an open medical-specialist model and 14.92% for a widely used general-purpose ASR model.

On general English, LibriSpeech test-clean WER increases slightly from 1.54% to 1.69%. The medical specialist remains better on exact matches for brand-name drugs and diagnostics. The study identifies real recordings of clinicians saying brand names as a possible next step.

For a new domain, the study recommends retaining at least 100 real, labeled clips for checkpoint selection and evaluating the entity types that matter to the application alongside general-speech performance.

The NeMo Speech fine-tuning tutorials cover training recipes, resource requirements, export, evaluation, new-language ASR, and code-switching ASR. ModelScope supports fine-tuning Nemotron 3.5 Transcribe through the ms-swift framework.

Access and deployment options

Nemotron 3.5 Transcribe checkpoints and interactive demonstrations are available through Hugging Face and ModelScope. Self-hosted deployment is supported through vLLM and SGLang-Omni, while hosted endpoints are available from Baseten and DeepInfra.

Granary v2 manifests are available through Hugging Face and ModelScope. The Granary-HiFi subset is also available through both platforms, with a Hugging Face demo.

Relevant repositories and resources include:

• NVIDIA NeMo Speech.
• The NeMo Curator Granary v2 pipeline.
• The vLLM deployment guide.
• The NeMo Speech fine-tuning guide.
• The fine-tuning agent skill.

These resources provide the model files, data manifests, serving instructions, and adaptation workflows needed to evaluate or specialize the system for a particular speech application.