SuTRA: Structurally-Unified Tokenization with Root Awareness

SuTRA：具有词根感知能力的结构化统一分词技术

Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. 现有的子词分词器（subword tokenizers）虽然优化了统计压缩，但忽略了形态结构，特别是词根与词缀之间的关系。这对形态丰富的印度语言来说是有害的，因为这些语言的基本单位是复杂的书写音节（aksharas），而非单个字母。

Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. 基于频率的方法往往会过度切分单词，随意地将词根和词缀拆开——我们将这种现象称为“形态破碎”（Morphological Shattering）。我们提出了 SuTRA（具有词根感知能力的结构化统一分词技术），这是一种具备形态感知能力的算法，它能够保持音节（akshara）的完整性，并对跨越形态边界的合并操作进行惩罚。

We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation. 我们还发布了一个针对印地语、马拉地语和古吉拉特语的全新形态分割数据集。SuTRA 有效减少了形态破碎现象，在形态对齐（边界 F1 值）方面实现了最高 +14.7% 的提升，在语义可恢复性（印地语）方面较 BPE 提升了 +34%。这些结构上的改进使机器翻译的 chrF2 指标平均提高了 +8.08。