Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

扩展印度语言预训练：2.8 万亿个标记与开放数据生成流水线

Native-script data for 22 languages, romanized data for twelve, and a reusable GPU-accelerated pipeline for translation, quality assessment, and repair. Authors: AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan. NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya.

包含 22 种语言的母语脚本数据、12 种语言的罗马化数据，以及一套可重复使用的、用于翻译、质量评估和修复的 GPU 加速流水线。作者：AI4Bharat — Mohammed Safi Ur Rahman Khan, Mitesh Khapra, Anoop Kunchukuttan。NVIDIA — Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen, Utkarsh Vaidya。

We’re excited to introduce Nemotron-pretraining-Indic-v1 approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it. These are the 22 languages listed in India’s Constitution: the first languages for 97% of Indians, or nearly 1.2 billion citizens at the 2011 Census.

我们很高兴推出 Nemotron-pretraining-Indic-v1，它涵盖了 22 种印度语言，约有 2.8 万亿个标记，并将与用于创建它的开放式 GPU 加速数据生成流水线一起发布。这些是印度宪法中列出的 22 种语言：它们是 97% 的印度人的第一语言，根据 2011 年的人口普查，这相当于近 12 亿公民。

Developed by NVIDIA with IIT Madras, India’s premier engineering institute, and its AI4Bharat lab, this release gives model builders a substantial new resource for Indic pretraining. The pipeline captures the full generation recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their own source material and generate more data for their needs.

该版本由 NVIDIA 与印度顶级工程学院印度理工学院马德拉斯分校（IIT Madras）及其 AI4Bharat 实验室共同开发，为模型构建者提供了用于印度语言预训练的重要新资源。该流水线涵盖了完整的生成方案，包括提示词、大语言模型工作流和质量检查，因此开发人员可以根据自己的源材料进行调整，并根据需要生成更多数据。

AI4Bharat, IIT Madras’ Indian language research lab, builds open-source datasets and models for Indian languages. Its resources, such as Sangraha, an open pretraining corpus for 22 Indian languages, have helped lay the foundation for open Indic language-model research. Our joint open-source release represents a substantial expansion of the open synthetic Indic training data that will be available to the community.

IIT Madras 的印度语言研究实验室 AI4Bharat 致力于构建印度语言的开源数据集和模型。其资源（例如用于 22 种印度语言的开放预训练语料库 Sangraha）已为开放的印度语言模型研究奠定了基础。我们联合发布的开源成果代表了社区可用的开放合成印度语言训练数据的重大扩展。

It brings a broad mix of curated English material, including mathematics, code, and question-answer content, into Indian languages, giving teams more training material for building models that can explain technical concepts, work through problems, and answer questions in those languages. Every record traces back to a curated source document, and each output is checked against its input, so the synthetic text stays grounded in real information.

它将包括数学、代码和问答内容在内的广泛精选英语材料引入印度语言，为团队提供了更多的训练材料，以构建能够用这些语言解释技术概念、解决问题并回答问题的模型。每条记录都可以追溯到精选的源文档，并且每个输出都经过输入核对，因此合成文本始终基于真实信息。

Mitesh Khapra, who heads AI4Bharat and made the 2025 TIME100 AI list, has named the core problem: Indian language technology trails English because “we do not have enough data for Indian languages.” We’re answering that call with more data, plus an open recipe for generating more.

领导 AI4Bharat 并入选 2025 年《时代》百大 AI 人物榜单的 Mitesh Khapra 指出了核心问题：印度语言技术落后于英语，因为“我们没有足够的印度语言数据”。我们正在通过提供更多数据以及生成更多数据的开放方案来响应这一需求。

The corpus also accounts for how people write. It includes native-script data for all 22 languages and romanized text for twelve, spanning 34 language-script configurations. Our workflows combine document-level translation, quality assessment, and targeted repair, with checks to preserve content such as equations, tables, and code. This data complements naturally authored Indic text, which remains essential for local knowledge and cultural context.

该语料库还考虑了人们的写作方式。它包括所有 22 种语言的母语脚本数据和 12 种语言的罗马化文本，涵盖 34 种语言脚本配置。我们的工作流结合了文档级翻译、质量评估和针对性修复，并带有检查机制以保留方程、表格和代码等内容。这些数据是对自然创作的印度语言文本的补充，而后者对于本地知识和文化背景仍然至关重要。

We’re sharing the open generation recipe so the community can not just reproduce the data but take this work further. Teams can use the corpus directly, adapt the prompts and quality checks, or bring their own source documents to the workflow. By opening up both the data and the process, we aim to make large-scale Indic data generation more accessible and enable more developers to contribute to stronger models across Indian languages.

我们分享开放的生成方案，以便社区不仅可以复现这些数据，还能进一步推进这项工作。团队可以直接使用该语料库、调整提示词和质量检查，或者将他们自己的源文档引入工作流。通过开放数据和流程，我们的目标是让大规模印度语言数据生成变得更容易，并使更多的开发人员能够为构建更强大的印度语言模型做出贡献。

Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens, drawn from a twelve-language subset of this corpus and natural Indic text. Our technical report on the model covers the full story. We plan to build on this corpus in future Nemotron models.

Nemotron 3.5 Super VL 的持续预训练增加了 100 亿个印度语言标记，这些标记取自该语料库的 12 种语言子集和自然印度语言文本。我们关于该模型的技术报告涵盖了完整的故事。我们计划在未来的 Nemotron 模型中继续利用该语料库。

What is in the corpus? Figure 1. The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Counts use the Gemma-4 tokenizer. Document records include translations and script variants of shared sources; they are not counts of unique English documents.

语料库中包含什么？图 1。该语料库包含 26,010 亿个母语脚本标记和 2,246 亿个罗马化标记。计数使用 Gemma-4 分词器。文档记录包括共享源的翻译和脚本变体；它们不是唯一英语文档的计数。

The corpus has two groups of language configurations: Native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu. Native-script data: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.

该语料库有两组语言配置：母语脚本和罗马化数据：孟加拉语、古吉拉特语、印地语、卡纳达语、马拉雅拉姆语、马拉地语、尼泊尔语、奥里亚语、旁遮普语、泰米尔语、泰卢固语和乌尔都语。母语脚本数据：阿萨姆语、博多语、多格拉语、克什米尔语、孔卡尼语、迈蒂利语、曼尼普尔语、梵语、桑塔利语和信德语。

Here, native-script describes the writing system: the text is still translated from English. Romanized means Indic-language text written in Latin script, not an English translation. For each of the twelve dual-script languages, romanization covers a subset of the accepted native-script translations. This distinction matters when constructing a training mixture. Native and romanized records offer different script exposure, but can express the same underlying content.

此处，“母语脚本”描述的是书写系统：文本仍然是从英语翻译而来的。“罗马化”是指用拉丁字母书写的印度语言文本，而不是英语翻译。对于 12 种双脚本语言中的每一种，罗马化涵盖了已接受的母语脚本翻译的一个子集。在构建训练混合集时，这种区别很重要。母语和罗马化记录提供了不同的脚本曝光，但可以表达相同的底层内容。

The token counts describe the retained corpus, including accepted rewrites, rather than a training run’s token budget. Counts also depend on the tokenizer used, so corpus inventory and training-token counts should be read separately.

标记计数描述的是保留的语料库，包括已接受的重写内容，而不是训练运行的标记预算。计数还取决于所使用的分词器，因此语料库清单和训练标记计数应分开阅读。

From English source documents to Indic training data To build this corpus, we start with documents from 15 Nemotron-CC-v2 subsets, covering web text, question-answer material, mathematics, code, and other specialized content. We use NeMo Data Designer, NVIDIA’s open-source framework for generating synthetic data, to run translation, quality checks, and targeted repair.

从英语源文档到印度语言训练数据：为了构建该语料库，我们从 15 个 Nemotron-CC-v2 子集的文档开始，涵盖网络文本、问答材料、数学、代码和其他专业内容。我们使用 NVIDIA 的开源合成数据生成框架 NeMo Data Designer 来运行翻译、质量检查和针对性修复。

Throughout the workflow, we retain each source identifier so developers can trace an output back to its original document. Clear provenance like this is part of what makes open data practical to audit and build on. Figure 2 shows the workflow for twelve languages. Figure 2. Four-stage Indic synthetic-data pipeline for twelve languages, implemented using NeMo Data Designer and Gemma-4-31B-it. We translate English documents, assess

在整个工作流中，我们保留了每个源标识符，以便开发人员可以将输出追溯到其原始文档。这种清晰的来源记录是使开放数据易于审计和构建的原因之一。图 2 显示了 12 种语言的工作流。图 2。使用 NeMo Data Designer 和 Gemma-4-31B-it 实现的 12 种语言的四阶段印度语言合成数据流水线。我们翻译英语文档，评估