The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
The Functionalizer：用于子词分词的无损函数分解
Abstract: Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and Héllo) as unrelated vocabulary entries, which fragments the embedding space, or discard this variation through lossy normalization.
摘要： 标准的子词分词器要么将单词的每一种拼写变体（例如 hello、Hello、HELLO 和 Héllo）视为不相关的词汇条目，这会碎片化嵌入空间；要么通过有损归一化丢弃这些变体。
We present the Functionalizer, a lossless pre-tokenizer framework that factors orthographic and structural variations into a compositional opcode/operand prefix stream before tokenization: a canonical base token (operand) prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area.
我们提出了 Functionalizer，这是一个无损的预分词框架。它在分词前将拼写和结构变体分解为组合式的操作码/操作数前缀流：即由编码在 Unicode 私有使用区（Private Use Area）中的参数化转换算子（操作码）作为前缀，修饰一个规范的基础词元（操作数）。
We introduce operators covering casing (CAPITALIZE), diacritics (13 dedicated opcodes), and character repetition (REPEAT, MULTIREPEAT), which are fully reversible.
我们引入了涵盖大小写（CAPITALIZE）、变音符号（13 个专用操作码）和字符重复（REPEAT, MULTIREPEAT）的算子，这些算子是完全可逆的。
Across six natural language and code corpora, the Functionalizer enables complete corpus coverage with significantly smaller vocabularies under unconstrained conditions, reducing actual vocabulary slot requirements by up to 16%.
在六个自然语言和代码语料库中，Functionalizer 在无约束条件下实现了完整的语料覆盖，且词汇表显著更小，将实际所需的词汇槽位减少了多达 16%。
When looking at sequence lengths, we observe a sharp domain-dependent tradeoff: it compresses indentation-heavy code sequences but inflates natural-language prose sequences.
在序列长度方面，我们观察到一种明显的领域依赖权衡：它压缩了缩进密集的代码序列，但会膨胀自然语言文本序列。
Preliminary downstream evaluations on 25M parameter GPT-2 scale models show that at this scale, the Functionalizer drastically improves code syntax validity and improves code character perplexity while maintaining similar text coherence on prose.
在 2500 万参数 GPT-2 规模模型上的初步下游评估显示，在此规模下，Functionalizer 在保持散文文本连贯性相似的同时，显著提高了代码语法的有效性并改善了代码字符困惑度。
These findings demonstrate that functional decomposition can be an effective mechanism for vocabulary-efficient, structurally aware language modeling, and motivate further validation at production scale.
这些发现表明，函数分解可以成为一种实现词汇高效、具备结构感知能力的语言建模的有效机制，并推动了在生产规模下进行进一步验证的需求。