Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems
Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems
超越每个字母两个字节：西里尔文 AI 系统中的分词开销
Abstract: Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disparities in cost and context capacity. We quantify this overhead across nine production tokenizers and five languages with standardized Cyrillic and Latin representations, covering 8.37 million word forms.
摘要： 现代多语言分词器在处理乌克兰语及其他代表性不足的西里尔字母语言时，往往比处理英语时产生更严重的碎片化，从而导致成本和上下文容量上的差异。我们针对九种生产级分词器和五种具有标准化西里尔字母和拉丁字母表示的语言，量化了这一开销，涵盖了 837 万个词形。
On a corpus benchmark, Ukrainian shows 68-121% token overhead on modern tokenizers and 220% on the older cl100k, measured through full-text fertility on the BrUK and Brown corpora. Overhead is negatively associated with Cyrillic vocabulary allocation in the subset with independently verified English baselines, although the association is not statistically significant (Spearman rho = -0.536, p = 0.215, n = 7).
在语料库基准测试中，通过 BrUK 和 Brown 语料库的全文本“生育率”（fertility）测量显示，乌克兰语在现代分词器上的 Token 开销为 68-121%，而在较旧的 cl100k 分词器上则高达 220%。在具有独立验证英语基准的子集中，开销与西里尔词汇分配呈负相关，尽管这种相关性在统计学上并不显著（Spearman rho = -0.536, p = 0.215, n = 7）。
We evaluate two mitigation strategies. LLMLingua-2 reduces Ukrainian input length by 47-49% on an e-commerce RAG benchmark of 1,536 products and 145 queries, with no compression-induced value losses among 80 retrievable cases. A balanced byte-level BPE tokenizer trained with a 200K vocabulary cap, converging at 158,184 actual entries, reduces the held-out UK/EN ratio from 2.22x to 1.30x.
我们评估了两种缓解策略。LLMLingua-2 在包含 1,536 个产品和 145 个查询的电子商务 RAG 基准测试中，将乌克兰语输入长度减少了 47-49%，且在 80 个可检索案例中未出现压缩导致的价值损失。一种以 20 万词汇量上限训练、最终收敛于 158,184 个条目的平衡字节级 BPE 分词器，将留存测试集中的乌克兰语/英语 Token 比率从 2.22 倍降低至 1.30 倍。
Romanization increases Ukrainian token counts by 2-19% on most tokenizers. Across the five languages, tokenization efficiency favors the script more prevalent in web data. These findings indicate that training data allocation contributes to Cyrillic tokenization overhead and that mitigation is possible at both inference and tokenizer-design stages.
在大多数分词器上，罗马化（转写）会使乌克兰语的 Token 数量增加 2-19%。在五种语言中，分词效率更倾向于在网络数据中更常见的书写系统。这些发现表明，训练数据的分配是导致西里尔文分词开销的原因之一，并且在推理阶段和分词器设计阶段均可进行优化缓解。