Scaling Point-in-Time Language Models

扩展“时间点”语言模型

Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.

摘要： 在不受限制的互联网语料库上训练的大型语言模型不可避免地会嵌入来自未来的信息，从而引入“前瞻偏差”（lookahead bias），这会损害金融和社会科学领域回测和因果推断的有效性。

Point-in-time language models—trained exclusively on text available up to each calendar date—eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts.

“时间点”语言模型（Point-in-time language models）仅使用截至每个日历日期前可用的文本进行训练，从构建机制上消除了这种信息泄露。然而，现有的研究成果通常会产生性能明显落后于不受限制模型的模型。

We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024.

我们证明，通过扩展规模可以显著缩小这一性能差距。我们使用来自 FineWeb 的 1 万亿个经过时间顺序过滤的 Token，训练了参数量高达 40 亿的仅解码器（decoder-only）Transformer 模型，并构建了一系列涵盖 2013 年至 2024 年的月度模型检查点。

Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks.

在一系列常识推理和语言理解基准测试中，我们的模型性能已接近在时间不受限数据上训练的同等规模领先开源模型（如 Gemma-3-4B 和 LLaMA-7B），尽管在某些任务上仍存在性能差距。

Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline—including dataset construction, training infrastructure, and evaluation code—to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

通过 LoRA 进行指令微调进一步提升了下游的可用性。我们发布了完整的流程——包括数据集构建、训练基础设施和评估代码——旨在实现可复现的“时间点”语言建模，并支持需要严格时间有效性的研究应用。