Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement

数据高效语言建模：从前沿进展到原则指导的模型改进

Abstract: Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. 摘要： 从有限的文本中学习要求模型能够利用上下文、泛化到新输入并保留有用的能力。求是引擎（Qiushi Engine）在 BabyLM 2026 Strict-Small 任务上开展了一项长周期、端到端的自主研究计划，语料库规模限制在 1000 万词以内，累计词展示量为 1 亿次。

Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. 研究分为三个阶段，连接了前沿进展、原则发现和原则指导下的模型改进。第一阶段结合了紧凑重述、预算再投资和残差增量学习，构建了一个前沿模型。

Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. 第二阶段发现，精确重复和对齐重述会产生不同的上下文使用模式，这取决于目标关系和预测窗口。在受控任务中，恢复熟悉的性能并不能确保未见过的输入仍能使用已学习的计算。这些发现支持了一个可测试的数据高效学习原则：围绕预测所需的上下文依赖关系组织经验；分别设计可见信息、监督和保留机制；并测试学习、泛化和保留能力。

Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. 第三阶段保留了源文本，掩盖了更多的局部线索，监督了选定的目标，并保留了对通常被掩盖输入的预测。来自同一父模型的两个延续种子在全部九项指标的综合表现上优于普通延续。两代模型之间的综合得分从 42.02 上升至 42.25；第二代模型在 2026 年 9 月 8 日的公开 Strict-Small 快照中取得了最高的综合得分。

Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. 后续研究探讨了压缩、关系锚点、共享表示和度量方法。模型已在 Hugging Face 上发布；代码和研究记录已同步至 GitHub 仓库。

Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them. 总之，这些阶段展示了研究 RSI（Research RSI）：即研究过程的递归自我改进。科学理解和方法创新改变了后续的问题和设计；新的实验则对这些问题和设计进行测试与完善。