Open ASR Leaderboard Adds Hindi and Indian English Evaluation

Open ASR Leaderboard Adds Hindi and Indian English Evaluation

Published August 28, 2026

Voice Arena and Hugging Face have added two evaluation sets for Hindi and Indian English to the Open ASR Leaderboard. The release introduces the leaderboard's first Global South language and expands its evaluation beyond the European languages previously included in its multilingual coverage.

The new datasets are Monsoon en-IN and Monsoon hi-IN. Hindi is spoken by more than half a billion people. Both datasets include public splits for self-scoring and private splits intended to reduce benchmark-specific optimisation. Across the four splits, the data covers 4,888 speaker-disjoint contributors and records 12 attributes for each speaker.

Why speaker coverage matters

Word Error Rate, or WER, remains a useful summary metric, but it does not show how recognition quality varies among different groups of speakers. Previous research has reported disparities in automatic speech recognition by race, gender, age, and accent. Conventional test sets often record what was said, but little about who said it.

The Monsoon collection was designed to expose those differences. It varies across nine dimensions:

• Geography
• Age
• Gender
• Vocabulary
• Devices
• Acoustic environments
• Speech type
• Speech rate
• Multiple valid transcripts for the same audio

These dimensions make it possible to examine cases where an aggregate WER appears stable while performance differs for a particular population.

Dataset composition

The four datasets were collected through one pipeline.

The recordings come from unscripted, two-person conversations captured on dual channels. Clips were segmented from individual channels so that each contains one speaker. In addition to the table's fields, each clip includes occupation, education, marital status, income band, handset brand, current city, and years spent in the current district.

The Indian English sets use standard string references, with normalisation to reduce the impact of spelling variation. Hindi uses a lattice of accepted transcript variants because a single spoken phrase can have several valid written forms.

Speaker coverage and diversity

Monsoon contains relatively few hours of audio but a large number of speakers. This distribution is intentional.

The distribution has several consequences:

• The ten largest contributors account for 2.8% to 6.8% of total duration, and more than half of the speakers appear exactly once. Results therefore represent averages across hundreds of voices rather than extended recordings from a small speaker group.
• The Indian English public split includes speakers from 428 native districts across 30 states and union territories. The Hindi sets cover 202 and 295 districts. Recordings use between 315 and 582 distinct device models, and no single model accounts for more than 2.1% of segments in any subset.
• Indian English includes speakers from all six geographic zones. In the public set, 35% of segments come from the South, 18% from the East, 18% from Central India, 16% from the North, and 11% from the West.

Metadata fields

Each Monsoon segment includes 18 columns, including 12 metadata fields.

The Hindi sets are concentrated in the Hindi belt, with Uttar Pradesh accounting for roughly 40% of speakers. The Indian English sets have a flatter geographic distribution: no state contributes more than 13%, and one-third of speakers come from outside the eight largest states. The public and private portions have closely matched distributions.

District and state information can provide useful accent signals because Indian administrative boundaries were often drawn along linguistic lines. Earlier analysis of Indian ASR on a closed benchmark reported district-level error rates ranging from roughly 4% to 44%, as well as differences associated with audio quality, speaking rate, utterance duration, gender, age, and device. Monsoon makes similar analysis possible on a public leaderboard test set.

Collection and quality control

Broad geographic coverage required recruiting across hundreds of districts rather than recording longer sessions with fewer participants. Distributed collection also introduced risks such as task gaming, playback audio, and inattentive annotation. The project used several checks to address these issues.

Recruitment and recording

Contributors were recruited through the Voice Arena community, whose reach includes rural and semi-urban districts. Pairs recorded two-person conversations through a peer-to-peer interface, using assigned everyday topics, their own handsets, and their own network connections.

Prospective contributors completed language-proficiency screening before receiving recording access. They were compensated and provided informed consent covering training use and distribution. Per-speaker duration caps, calibrated for each language's population size and geographic distribution, limited the influence of prolific contributors. More than half of the speakers contributed exactly one segment.

Eliciting spontaneous speech

Short, sparse responses can be difficult to collect at scale, so conversations began with open-ended narrative prompts followed by progressively revealed questions. Topics included travel, healthcare, agriculture, education, and digital services. The prompts encouraged extended descriptions without scripting the responses.

Candidate topics were generated with large language models and then reviewed and localised by native-speaker linguists.

Quality checks

Before transcription, recordings passed several gates:

• Language identification models, trained on human-annotated data covering more than 30 languages, verified the assigned language.
• A dedicated classifier corroborated self-reported speaker gender without replacing the self-report.
• Another model distinguished genuine spontaneous conversation from prerecorded or played-back audio.
• Signal-to-noise estimation removed recordings degraded beyond intelligibility, while natural environmental noise was retained.
• Voice activity detection segmented the recordings at two seconds of continuous silence or at a 15-second soft cap ending at the next detected silence.
• Segmentation was performed independently for each channel, producing single-speaker, single-channel segments.
• Segments were checked with DNSMOS P.808.

Transcription

Reference transcripts were produced by human annotators. An initial draft came from internal ASR models trained on in-domain data. Those models do not appear on a public leaderboard, so no system evaluated on Monsoon contributed to the references used for scoring.

Native-speaking linguists then followed a five-level protocol with separate correction and verification stages. Each correction round was independently checked by another annotator. The process addressed ambiguous phonetic realisations, code-switching boundaries, named entities, and orthographic consistency across spelling variants. Numerals were written as words so the transcript corresponded directly to the spoken form.

Segments still flagged at the final level were retranscribed before admission. Automated monitoring identified submissions with characters outside the target script, unnatural character or word repetition, and unusually low or high edit counts.

Regional variation in Indian English

The public Indian English split illustrates the analysis enabled by the metadata. Eight leaderboard models produced WER scores between 4.81 and 4.99, a difference of 0.18 points from best to worst.

Grouping speakers by region produced a wider spread. Native districts were grouped into five well-sampled zones using the Ministry of Home Affairs classification of Indian states. openai/whisper-large-v3-turbo varied by 0.46 points across the zones. mistralai/Voxtral-Mini-3B-2507, which was 0.14 points behind it on the overall corpus, varied by 1.68 points, scoring 4.38 in the Central zone and 6.06 in the East.

The hardest region also differed by model. ibm-granite/granite-speech-3.3-2b performed worst in the North, microsoft/VibeVoice-ASR-HF in the South, and mistralai/Voxtral-Mini-3B-2507 in the East. If one region were inherently harder for all systems, the models would rank the zones similarly. Their different patterns indicate that regional performance depends partly on the systems themselves.

Region is one of 12 recorded attributes. The same type of breakdown can be applied to age, education, occupation, and handset, using the released metadata.

Orthographic variation in Hindi

English spelling variation is often limited enough for a normaliser to map common differences, such as British and American spelling, punctuation, case, and digits versus words. Hindi presents a more open-ended problem. Everyday speech often includes code-switching, English-origin terms do not have settled Devanagari spellings, and compound words may be written together or separately.

A phrase can therefore have ten or more valid written forms. With a single reference, WER may reward a system for reproducing the annotator's chosen spelling even when another system recognised the audio equally well.

The Hindi datasets use a transcript lattice. For each transcript span, the lattice lists the written forms accepted as correct. Candidate variants were drawn from multiple ASR transcripts of the same audio and expanded with language models. Native-speaking linguists then determined which variants were valid for the utterance and removed the rest.

Hindi results use Orthographically-Informed Word Error Rate, or OIWER, instead of WER. Introduced by AI4Bharat, OIWER aligns a hypothesis against the accepted forms for each span. Any admitted form is counted as correct, while genuine recognition errors are charged.

The same hypotheses were scored against both a flattened, single-string reference and the full lattice. Error rates increased for every system when scored against the flattened reference, but not uniformly, and model rankings changed. This demonstrates how a single reference can measure orthographic choice in addition to recognition quality.

The implementation is available as voi-oiwer, allowing results on these datasets to be reproduced.

How models are evaluated

For private splits, models are submitted to the Open ASR Leaderboard through its GitHub repository. The process is:

1. Open a pull request using the model checklist and report results on the public datasets.
2. The public results are verified, and metrics are computed on the private splits.
3. The submitter confirms the resulting scores.

Indian English appears on the main leaderboard as Voice Arena Monsoon in the default column set. It contributes to the headline Average WER for every model. Its private split is included in the aggregated Private (conversational) column alongside Appen and DataoceanAI data.

The public and private Hindi results appear in the leaderboard's Multilingual tab. A model is ranked there only when it supports every selected language, enabling like-for-like comparison. Hindi can also be selected from the Language dataset breakdown menu.

Broader implications

Hindi illustrates a wider evaluation challenge. Languages with multiple valid writing systems or spelling conventions, and languages spoken by populations missing from existing benchmarks, can expose both recognition and representation gaps.

The Monsoon sets do not eliminate those gaps. They provide a way to examine them on a leaderboard test set, with speaker metadata and reference alternatives that help distinguish differences related to who is speaking from differences related to how the language is written.

The four datasets are part of Monsoon, Voice Arena's broader dataset initiative for the Global South.