NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
NCP-ArchPreview 技术报告：通过“下一概念预测”迈向潜在空间语言模型
Abstract: We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation.
摘要： 我们推出了 NCP-ArchPreview，这是一种潜在空间语言模型，它将自回归预训练推向了超越标准“下一词元预测”（NTP）的范畴。除了 NTP 之外，该模型还通过“下一概念预测”（NCP）进行学习，以预测跨越多个词元的离散概念。在保留标准词元级自回归生成的同时，引入了一个更明确且更具挑战性的概念级目标。
NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end.
NCP-ArchPreview 通过直接从其隐藏状态构建乘积量化概念词表来建立潜在空间，随后通过专门的概念模块学习预测未来概念。这些预测出的概念被反馈回词元层以指导后续生成，NTP 和 NCP 任务以端到端的方式联合训练。
We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space language model to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99-point gain on GSM8K.
我们将该架构扩展至 89 亿参数，并在 Dolma-3 数据集的 5.73 万亿词元上进行训练，这是迄今为止潜在空间语言模型规模最大的一次演示。值得注意的是，NCP-ArchPreview 仅消耗了总训练词元的 51.3%，就达到了 OLMo-3-7B 的最终预训练损失水平。在完成完整预训练后，其下游任务宏平均得分比 OLMo-3-7B 高出 2.45 分，其中在 GSM8K 任务上取得了 5.99 分的显著提升。
Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective. Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline.
对照实验清晰地分离并展示了源自潜在架构和 NCP 目标的性能提升。此外，NCP-ArchPreview 仅使用标准计算量的 85%，就接近了参数严格对齐的 89 亿基准模型的训练损失。
The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.
所学习到的潜在空间在预训练阶段结束后依然极具价值：仅更新 1700 万参数的 VQ 模块即可为领域适应提供一种新颖且轻量级的接口；同时，将概念表示简单注入到 DFlash2 草稿模型中，可以在几乎不增加额外开销的情况下，将平均接受长度提升 4.17%。