Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch

Computer Science > Computation and Language arXiv:2610.10592 (cs) [Submitted on 6 Oct 2026] Title: Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch Authors: Szymon Kocur.

计算机科学 > 计算与语言 arXiv:2610.10592 (cs) [提交于 2026 年 10 月 6 日] 标题：Wieszcz-XIX：一个 31 亿词的 1918 年前波兰语语料库及从零训练的时间受限语言模型 作者：Szymon Kocur。

Abstract: Historical Polish is well documented as a language but annotated in machine-readable form only to about a million words for the period this paper covers; the rest sits behind optical character recognition of variable quality. We present Wieszcz-XIX, a corpus of 6.75 billion tokens (about 3.1 billion words) in 294,369 documents, most of them periodical issues, of Polish published from 1800 to 1918, assembled from Wolne Lektury and the Internet Archive by a pipeline that filters, deduplicates, audits for post-1918 leakage and splits at the document level.

摘要：历史波兰语作为一种语言有很好的文献记录，但在本文涵盖的时期内，仅有约一百万词以机器可读的形式进行了标注；其余部分则受限于质量参差不齐的光学字符识别（OCR）。我们提出了 Wieszcz-XIX，这是一个包含 67.5 亿个标记（约 31 亿词）的语料库，涵盖了 294,369 份文档（主要是期刊），这些文档均为 1800 年至 1918 年间出版的波兰语资料。该语料库通过一个流水线从 Wolne Lektury 和互联网档案馆汇编而成，该流水线执行了过滤、去重、针对 1918 年后数据泄露的审计以及文档级拆分。

It is over three orders of magnitude larger than the annotated corpus of the same period, and we quantify its defects: recognition corruption against a false-positive floor, near-identical duplication, which is removed, and post-1918 leakage, which is excluded from the training corpus itself down to a known residue of 0.04 to 0.38% of its bytes, found in the transcribed source, so the published corpus is the trained one document for document.

它比同一时期的标注语料库大三个数量级以上，我们量化了其缺陷：针对误报底线的识别损坏、近乎完全重复的内容（已被移除），以及 1918 年后的数据泄露（已从训练语料库中排除，仅保留了转录源中 0.04% 到 0.38% 的已知残留字节），因此发布的语料库与训练所用的文档一一对应。

On a hand-corrected sample the character error rate is 0.68% where the text is legible, and 45% of the sampled passages cannot be corrected. On it we train a ladder of decoder-only models from 47M to 349M parameters from scratch, and measure their temporal boundedness.

在人工校对的样本中，可读文本的字符错误率为 0.68%，而 45% 的采样段落无法校对。在此基础上，我们从零开始训练了一系列参数规模从 4700 万到 3.49 亿不等的仅解码器（decoder-only）模型，并测量了它们的时间受限性。

Against two modern Polish base models, one far larger, the 349M shows a crossover, as does the 107M against the comparator of its size: post-1918 vocabulary costs them about 3.1 bits per byte more than period vocabulary, a gap the comparators do not show, and period vocabulary costs them fewer bits than it costs the comparators.

与两个现代波兰语基础模型（其中一个规模大得多）相比，349M 模型表现出了交叉点，107M 模型与其同等规模的对比模型相比也表现出了同样的情况：1918 年后的词汇对它们来说比时期词汇每字节多消耗约 3.1 比特，这是对比模型所没有的差距，且时期词汇对它们而言比对对比模型消耗的比特更少。

Shown period text, the models keep its spelling and the comparators only partly. Adding parameters gains about twice as much as a second pass over the data. We release the corpus, code and weights. Content warning: the models reproduce period prejudice, including antisemitic statements.

在展示时期文本时，这些模型保留了其拼写，而对比模型仅部分保留。增加参数带来的增益大约是数据二次遍历的两倍。我们发布了语料库、代码和权重。内容警告：这些模型重现了当时的偏见，包括反犹太主义言论。