Index SLM Technical Report

Abstract: We present Index-1.9B, a series of open small language models developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion non-embedding parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; Index-1.9B-Chat, aligned from the base model with supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for few-shot role-playing customization.

摘要： 我们介绍了由哔哩哔哩（Bilibili）开发的系列开源小语言模型 Index-1.9B。该系列包含四款模型：Index-1.9B-Base，一款拥有 19 亿非嵌入参数的基础模型，在 2.8 万亿个以中英文为主的 Token 上进行了预训练；Index-1.9B-Pure，一个对照变体，采用相同的训练方案，但严格过滤了语料库中所有类似指令的数据；Index-1.9B-Chat，通过监督微调（SFT）和直接偏好优化（DPO）从基础模型对齐而来；以及 Index-1.9B-Character，它通过检索增强生成（RAG）技术增强了聊天模型，以实现少样本的角色扮演定制。

Pre-training employs a Warmup-Stable-Decay learning-rate schedule in which the concentration of curated data is raised substantially during the decay phase, together with a Norm-Head output layer that stabilizes training under large learning rates. On a suite of standard benchmarks covering examination, reasoning, mathematics, and code, Index-1.9B-Base attains an average score of 64.92, competitive with or exceeding open models of several times its size.

预训练采用了“预热-稳定-衰减”（Warmup-Stable-Decay）的学习率调度策略，在衰减阶段大幅提高了精选数据的比例，并结合了 Norm-Head 输出层，以确保模型在大学习率下的训练稳定性。在涵盖考试、推理、数学和代码的一系列标准基准测试中，Index-1.9B-Base 取得了 64.92 的平均分，其表现足以媲美甚至超越规模为其数倍的开源模型。

We further report controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training, and we document an unexplained surge in benchmark performance midway through the constant-learning-rate phase. All models, together with evaluation code, are released at this [https URL].

我们进一步报告了关于模型深度、学习率大小与调度、学习率衰减与数据质量之间的相互作用，以及预训练中包含指令数据的影响的对照研究。此外，我们记录了在恒定学习率阶段中期，基准测试性能出现的一种无法解释的激增现象。所有模型及评估代码均已发布至此 [链接]。