Entity-Aware Evaluation of OCR Systems for Historical Document Transcription
Computer Science > Computation and Language arXiv:2610.33004 (cs) [Submitted on 7 Oct 2026] Title: Entity-Aware Evaluation of OCR Systems for Historical Document Transcription. Authors: Camilo Mariño (UDELAR, CB), Diego Belzarena (UDELAR, CB), Marina Gardella (CB), Ignacio Ramírez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong).
计算机科学 > 计算与语言 arXiv:2610.33004 (cs) [提交于 2026 年 10 月 7 日] 标题：历史文档转录中 OCR 系统的实体感知评估。作者：Camilo Mariño (UDELAR, CB), Diego Belzarena (UDELAR, CB), Marina Gardella (CB), Ignacio Ramírez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong)。
Abstract: Optical Character Recognition (OCR) is essential for digitizing historical archives, yet standard evaluation metrics such as Character Error Rate (CER) do not necessarily reflect whether meaningful information, such as names, dates, and identifiers, is preserved. In this work, we investigate entity-level analysis of OCR errors as a complementary perspective for evaluating OCR quality in historical documents.
摘要：光学字符识别（OCR）对于历史档案的数字化至关重要，但诸如字符错误率（CER）之类的标准评估指标并不一定能反映出姓名、日期和标识符等有意义的信息是否得到了保留。在这项工作中，我们研究了 OCR 错误的实体级分析，将其作为评估历史文档中 OCR 质量的一种补充视角。
Using a publicly available benchmark of 175 documents from the Berrutti Archive, augmented with 5,209 manually annotated entity mentions that will be released upon acceptance, we compare sequence-recognition OCR systems, OCR-specialized vision-language models, and general-purpose vision-language models.
我们使用了一个来自 Berrutti 档案库的包含 175 份文档的公开基准，并辅以 5,209 个手动标注的实体提及（将在论文被录用后发布），对比了序列识别 OCR 系统、OCR 专用视觉语言模型以及通用视觉语言模型。
Beyond global transcription accuracy, we analyze entity preservation and recoverability under varying character-level edit tolerances. Our experiments show that OCR rankings depend on the evaluation criterion: while Qwen 3.6 achieves the highest entity coverage across all tested tolerances, several systems with lower exact-match coverage recover substantially when small errors are tolerated, suggesting that many of their errors correspond to localized character-level deviations rather than fundamentally incorrect entity transcriptions.
除了全局转录准确率之外，我们还分析了在不同字符级编辑容差下的实体保留和可恢复性。实验表明，OCR 的排名取决于评估标准：虽然 Qwen 3.6 在所有测试容差下都实现了最高的实体覆盖率，但一些精确匹配覆盖率较低的系统在容忍小错误时表现出显著的恢复能力，这表明它们的许多错误对应于局部的字符级偏差，而非根本性的实体转录错误。
Conversely, some systems remain unable to recover many entities even under relaxed matching. These findings show that OCR quality cannot be fully characterized by aggregate transcription metrics alone and highlight the importance of evaluating how OCR errors affect the preservation and recoverability of meaningful information.
相反，一些系统即使在放宽匹配条件的情况下，仍然无法恢复许多实体。这些发现表明，仅凭汇总的转录指标无法全面表征 OCR 质量，并强调了评估 OCR 错误如何影响有意义信息的保留和可恢复性的重要性。