Scaling Indic Pretraining to 2.8 Trillion Tokens With an Open Data Pipeline

Scaling Indic Pretraining to 2.8 Trillion Tokens With an Open Data Pipeline

Published October 16, 2026

NVIDIA and IIT Madras's AI4Bharat lab have developed Nemotron-pretraining-Indic-v1, a corpus containing approximately 2.8 trillion tokens across 22 Indian languages. The dataset is accompanied by an open, GPU-accelerated pipeline for translation, quality assessment, script conversion, and targeted data repair.

The 22 languages are those listed in India's Constitution. Together, they represent the first languages of 97% of India's population, or nearly 1.2 billion people according to the 2011 Census.

The release is intended to provide model developers with a larger resource for Indic pretraining. Its pipeline documents prompts, large language model workflows, quality checks, and repair procedures, allowing developers to adapt the process to other source collections.

AI4Bharat develops open datasets and models for Indian languages. Its resources include Sangraha, an open pretraining corpus covering 22 Indian languages. The new release expands the amount of open synthetic Indic training data by translating curated English material covering mathematics, code, question-answer content, and other specialized subjects.

Each generated record retains a link to its source document, and outputs are checked against their inputs. This is intended to keep the synthetic text grounded in the original information while allowing models to explain technical concepts, solve problems, and answer questions in Indian languages.

Language and script coverage

The corpus includes native-script data for all 22 languages and romanized text for twelve, resulting in 34 language-script configurations. The data is organized into two groups:

• Native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.
• Native-script data: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.

Native-script data refers to writing systems used for Indic languages, although the underlying text was translated from English. Romanized data consists of Indic-language text written in the Latin alphabet, rather than English text translated into another language.

For the twelve languages with both forms, romanization covers a subset of accepted native-script translations. Native and romanized records provide different script exposure and can represent the same underlying content, so the distinction matters when designing a training mixture.

The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, for a combined total of approximately 2.8 trillion tokens. These counts use the Gemma-4 tokenizer. Because records can include translations and script variants of shared source documents, the totals do not represent unique English documents. Token counts also depend on the tokenizer and should be distinguished from a training run's token budget.

From English documents to Indic training data

The corpus was built from documents in 15 Nemotron-CC-v2 subsets. The source material covers web text, question-answer content, mathematics, code, and other specialized areas.

NVIDIA NeMo Data Designer runs the translation, assessment, and repair stages. Source identifiers remain attached throughout the workflow, allowing developers to trace an output to its original document.

The process has two main variants: one for twelve languages that receive both native-script and romanized data, and another for ten languages that receive native-script data only.

The twelve-language workflow

For the twelve dual-script languages, documents are translated in full so that models can use context across paragraphs, tables, and equations. The workflow uses Gemma-4-31B-it, selected by comparing models on complete documents and examining translation quality and preservation of technical content.

Prompts identify prose that should be translated and content that should be preserved, including code, equations, URLs, and formatting.

The same model assesses each translation against its English source using a rubric inspired by Multidimensional Quality Metrics. The rubric evaluates:

• Meaning
• Fluency
• Terminology
• Style
• Formatting

Only outputs scoring at least 75 out of 100 are retained. The score is a rubric result, not an accuracy percentage. Outputs below the threshold receive feedback-based repairs and are assessed again.

This process recovered 438.4 billion tokens across native-script and romanized data. Those tokens are already included in the corpus total. Sampled human reviews and tests containing deliberate errors were used to identify omissions and residual English and to improve prompts and quality checks.

Accepted native-script translations form the basis for romanized data. A subset is converted into Latin script and checked against the native-script version. This separates translation from script conversion. In an initial human comparison, the two-step method produced fewer hallucinated additions and less residual English than direct generation of romanized text from English.

The ten-language workflow

The remaining ten languages use a modified process designed to address long-document quality and script-stability issues. Translation is performed with Indic Translate, a document-level translation model for English and 22 Indian languages from Bodhan AI, a nonprofit AI education center incubated at IIT Madras.

Gemma-4-31B-it assesses each translation against the English source. The whole-document method is used for eight languages. Longer Santali documents are translated through overlapping windows and then reconstructed. For Manipuri, the workflow translates into Bengali script before converting the output into Meitei Mayek, its native script.

The workflow retains outputs scoring at least 75 out of 100. It produces native-script data only and does not use feedback-based rewriting.

Results from Indic continued pretraining

The Nemotron 3.5 Super VL technical report compares the checkpoint immediately before and after Indic continued pretraining (CPT).

The configured CPT budget was 100 billion training tokens. Indic data accounted for 10%, or 10 billion tokens, drawn from a mixture of synthetic and natural Indic text. The remaining 90% preserved the relative proportions of the preceding training mixture.

Scores increased on all five reported Indic knowledge and reasoning benchmarks:

• MILU
• Indic MMLU-Pro
• Indic ARC-Challenge
• Indic ARC-Easy
• Indic GSM8K

Reported gains ranged from 4.98 to 9.82 percentage points. The comparison covers a training stage combining synthetic and natural Indic data, so it does not isolate the contribution of the synthetic corpus. Language coverage differs by benchmark. GSM8K excludes Odia because evaluation exceeded the serving context limit.

The companion Indic report examines source mixtures, synthetic-data pool size, and script coverage using first-pass accepted data from the twelve-language group. Evaluating the complete corpus, including recovered rewrites and the other ten languages, remains a separate area for controlled study.

Broader capabilities in Nemotron 3.5 Super VL

The released Nemotron 3.5 Super VL model was also compared with the previous-generation Nemotron 3 Super. These results reflect the complete training pipeline, including Indic CPT and post-training.

Nemotron 3.5 Super VL scored higher on five of the six task benchmarks shown, covering knowledge, reasoning, mathematics, code, and translation. Its IFEval 1.5 instruction-following score decreased by 2.04 points.

The model also improved across all four IndiVibe dimensions:

• Usefulness
• Fluency
• Language and script fidelity
• Appropriate verbosity

IndiVibe is an open benchmark from Sarvam AI containing assistant-style prompts in Indian languages. Its results and the task benchmarks evaluate both model capabilities and communication quality in Indic languages. IndiVibe values are preliminary conditional-preference scores, not direct head-to-head win rates. WMT24++ changes are reported in chrF points rather than percentage points.

Working with the corpus

Developers can select the language and script configurations needed for a model, inspect sample records, and review the dataset card and license before training. Configurations use language, script, and region tags. For example:

• hi-Deva-IN represents Hindi in Devanagari.
• hi-Latn-IN represents romanized Hindi.

The following example discovers available splits and streams one sample without downloading the complete corpus:

from datasets import get_dataset_split_names, load_dataset

dataset_id = input("Repository ID from the dataset card: ").strip()
config = "hi-Deva-IN"

splits = get_dataset_split_names(dataset_id, config_name=config)
print("Available splits:", splits)

data = load_dataset(
    dataset_id, name=config, split=splits[0], streaming=True
)
print(next(iter(data)))

The Hugging Face Datasets streaming API supports record inspection without downloading the entire dataset. Dataset revisions should be pinned for reproducible experiments once the release is available.

For an initial training study, teams can compare mixtures at a fixed token budget while recording the proportions of:

• Natural and synthetic data
• Native-script and romanized data
• Each language

Evaluations should track per-language performance, English capability retention, and language and script quality. Held-out splits should account for records derived from the same source document across languages and scripts.

The corpus can also be combined with Nemotron post-training data, including the Chat & Instruction and Post-Training Blends collections. For India-specific synthetic data, Nemotron-Personas-India provides synthetic personas in English and Hindi grounded in real-world distributions for India.

Adapting the generation recipe

The open-source Nemotron Bharat translation plugin for NeMo Data Designer includes:

• Translation and evaluation prompts
• Optional feedback-based rewriting
• Document windowing
• Script-remapping support

Its configuration reference explains how to select models and languages, adapt prompts, and set rewrite parameters. An accompanying developer note describes the workflow and engineering decisions.

A practical adaptation process begins with a small sample of source documents. Translation quality and the preservation of code, mathematics, and formatting should be inspected before scaling. Native-language reviewers can help refine prompts and checks.

Generation and corpus acceptance are separate stages. The plugin returns candidate records and scores, while the filtering policy determines which records enter the training corpus.

Scope and limitations

Translation increases access to source material, but it does not replace text originally written in Indian languages or guarantee coverage of local knowledge and culture. The corpus should be combined with appropriately sourced natural Indic text and evaluated for its intended use.

Quality assessment is primarily automated and supplemented by sampled human review, whose coverage varies by language. In the twelve-language workflow, the translator also serves as the evaluator. Retained text may still contain errors, and the ten-language workflow requires broader native-speaker validation.

Source-ID checks remove processing duplicates, but they are not semantic deduplication of similar content.

AI4Bharat at IIT Madras, with support from NVIDIA, is also developing Indic-LLM-Eval-Suite, an open and evolving set of benchmarks for Indic model capabilities. The suite evaluates both task performance and language quality.

Further reading and acknowledgments

The companion reports, Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs and the Nemotron 3.5 Super VL and Lightning technical report, provide additional information about corpus accounting, generation methods, experiments, and evaluation.

The benchmark methodology is discussed in Better Models Need Better Benchmarks: Introducing Indic-LLM-Eval-Suite.

The work combines contributions from NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI. The pipeline also uses Google's Gemma, Bodhan AI's Indic Translate, and NeMo Data Designer.