Letter Lemmatization: One-to-one and Banded RNNs for Reversing Character-Set Simplification and Abbreviation in Medieval Text

字母词形还原：用于逆转中世纪文本字符集简化与缩写的“一对一”及“带状”循环神经网络

---

Abstract: Medieval document transcribers have very different practices; on top of that, heterogeneous digitization policies have resulted in corpora where the character-set must be viewed as fluid. In this paper we address the problem of changing between character-sets in a flexible manner. We focus on one-to-one character mappings and train character-level one-to-one RNNs to undo them with self-supervision; recovering half the CER even with 20 text lines. We analyse the use of these one-to-one networks for HTR post-correction and we see that they obtain significant improvements while totally ignoring ins-dels. We then use the exact same networks with character-level alignment groundtruth compiled from parallel corpora in a training and inference mode we call Banded RNNs. We use such networks to successfully expand abbreviations in medieval charter transcriptions. Finally we introduce an elaborate heuristic which takes the characters of two arbitrary character-sets and defines a metric encapsulating what we consider to be semantic similarity of characters. We call the construction of such mappings letter lemmatization and present a rich Python library that efficiently performs all presented methods.

摘要： 中世纪文献抄写员的实践方式各不相同；此外，异构的数字化政策导致语料库中的字符集必须被视为流动的。在本文中，我们以灵活的方式解决了字符集转换的问题。我们专注于一对一的字符映射，并训练字符级的一对一循环神经网络（RNN）通过自监督方式来逆转这些映射；即使仅使用 20 行文本，也能恢复一半的字符错误率（CER）。我们分析了这些一对一网络在手写文本识别（HTR）后校正中的应用，发现它们在完全忽略插入-删除（ins-dels）操作的情况下，仍能获得显著的改进。随后，我们使用完全相同的网络，结合从平行语料库中编译的字符级对齐基准数据，在一种我们称为“带状 RNN”（Banded RNNs）的训练和推理模式下进行操作。我们利用这些网络成功地扩展了中世纪宪章抄本中的缩写。最后，我们引入了一种精细的启发式方法，它提取任意两个字符集的字符，并定义了一种度量标准，用以概括我们所认为的字符语义相似性。我们将这种映射的构建过程称为“字母词形还原”（letter lemmatization），并提供了一个功能丰富的 Python 库，可以高效地执行文中提出的所有方法。