tokenizers v1: encode, decode and scaling, measured
tokenizers v1: encode, decode and scaling, measured
The tokenizer has not historically been the bottleneck within ML workflows. Compute-wise, tokenization is light compared to the heavy modeling happening in the rest of the pipeline. Yet, in some cases, it has rapidly become key to accelerating (or slowing down) your machine learning work. As models become faster and workloads scale, that balance begins to shift. 在机器学习工作流中，分词器（tokenizer）历来不是瓶颈。从计算角度来看，与流水线其余部分繁重的模型计算相比，分词过程非常轻量。然而，在某些情况下，它已迅速成为加速（或拖慢）机器学习工作的关键。随着模型运行速度的提升和工作负载的扩展，这种平衡开始发生变化。
Training on massive datasets, serving many concurrent requests, or repeatedly processing long inputs can put enough pressure on the tokenizer that it starves the model of data. This is why we have chosen to heavily focus on performance for the upcoming version 1 of tokenizers. Tokenization should be light and should scale with your workflow. Your GPUs should never sit idle waiting for the CPU to complete its tokenization. 在海量数据集上进行训练、处理大量并发请求或重复处理长输入，可能会给分词器带来巨大压力，导致模型因数据供应不足而“饥饿”。这就是为什么我们决定在即将发布的 tokenizers v1 版本中重点关注性能。分词过程应当轻量且能随工作流扩展。你的 GPU 绝不应在等待 CPU 完成分词时处于空闲状态。
In this article, we look at what makes v1 faster than v0.23, often by tens of times. This work was entirely possible thanks to the rest of the ecosystem. Tokenization is a very active area of open source work, and libraries such as gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper and ai-tokenizer, as well as many others, have each pushed on what a fast tokenizer can be. We read that work, and several of the ideas below reached us because another project showed they were worth trying. 在本文中，我们将探讨 v1 版本为何比 v0.23 快（通常快几十倍）。这项工作完全得益于整个生态系统的贡献。分词是开源领域非常活跃的板块，诸如 gigatoken、tiktoken、kitoken、tokie、fastokens、wordchipper 和 ai-tokenizer 等众多库，都在不断推动分词器性能的极限。我们参考了这些成果，下文提到的几个想法正是因为其他项目证明了其可行性，才被我们采纳。
Before this refactor, tokenizers was nowhere near the performance it could have had, so contributing to it may not have seemed worth it. With this refactor, we hope to make clear that we intend tokenizers to be a library worth contributing to. We also thank IBM, NVIDIA, and the ExecuTorch team for contributing patches and helping us test across a wide range of hardware to broaden platform support. 在这次重构之前，tokenizers 的性能远未达到应有的水平，因此向其贡献代码似乎并不值得。通过这次重构，我们希望明确一点：我们致力于将 tokenizers 打造成一个值得贡献的库。我们还要感谢 IBM、NVIDIA 和 ExecuTorch 团队贡献补丁，并帮助我们在各种硬件上进行测试，从而扩大了平台支持范围。
Results
结果
We showcase results for the release candidate of tokenizers v1 against other widely used alternatives. We go over single-threaded, multi-threaded, scaling across threads, per-model comparison, per-language comparison, latency, decoding throughput, memory heap, as well as crate size. We run this from the tokbench repository, and add a command to rerun the benchmarks on your hardware if you would like to do so. 我们展示了 tokenizers v1 发布候选版本与其他广泛使用的替代方案的对比结果。我们涵盖了单线程、多线程、多线程扩展性、各模型对比、各语言对比、延迟、解码吞吐量、内存堆占用以及 crate 大小等指标。我们通过 tokbench 仓库运行这些测试，并添加了一个命令，以便你可以在自己的硬件上重新运行这些基准测试。
What V1 Is
什么是 V1
v1 will produce the same token IDs as v0.23. The goal was to preserve the output, the API, the vocabulary and the merge ranks, and improve everything that can be improved. That includes breadth. The library stays general across tokenizer families rather than specialising on BPE, so v1 loads everything v0.23 loaded. v1 将产生与 v0.23 相同的 token ID。我们的目标是在保持输出、API、词汇表和合并优先级（merge ranks）不变的前提下，改进所有可以改进的地方，包括广度。该库在分词器家族中保持通用性，而不是专门针对 BPE，因此 v1 可以加载 v0.23 支持的所有内容。
A tokenizer converts text into the list of integers a model reads. tokenizers runs that conversion in four stages. Normalization applies operations such as lowercasing or Unicode normalization to the raw text. Pre-tokenization splits the text into smaller pieces called pre-tokens. The model turns each pre-token into tokens and maps them to IDs in its vocabulary. Post-processing adds any special tokens the model expects. 分词器将文本转换为模型可读取的整数列表。tokenizers 在四个阶段完成此转换：归一化（Normalization）对原始文本应用小写化或 Unicode 归一化等操作；预分词（Pre-tokenization）将文本拆分为较小的片段，称为预分词（pre-tokens）；模型将每个预分词转换为 token，并将其映射到词汇表中的 ID；后处理（Post-processing）添加模型所需的任何特殊 token。
The Split: Bitstreams Instead Of A Regex
拆分：使用位流（Bitstreams）代替正则表达式
BPE models use a regular expression to split the input text into smaller, easier to process chunks called pre-tokens. Merges happen inside a pre-token and never across the boundary between two of them, so this split decides what the rest of the pipeline sees. That regular expression is a fixed parameter of the model. It ships with the tokenizer and never changes at runtime, so there is no need for a general-purpose regex engine to interpret it on every encode. BPE 模型使用正则表达式将输入文本拆分为更小、更易于处理的块（即预分词）。合并操作发生在预分词内部，绝不会跨越两个预分词之间的边界，因此这种拆分决定了流水线后续部分的处理对象。该正则表达式是模型的一个固定参数。它随分词器一起发布，且在运行时从不改变，因此无需在每次编码时都使用通用的正则表达式引擎来解释它。
An equivalent splitting function can be written by hand, once, for the pattern a given model actually uses. A hand-written function can then use the SIMD instructions (single instruction, multiple data) of a modern CPU, which apply one operation to many bytes at once and suit UTF-8 text well. bitcannon views the input’s bytes as parallel streams of bits, so boundaries fall out of boolean operations across whole registers instead of a scan that advances one character at a time. 对于给定模型实际使用的模式，可以手动编写一个等效的拆分函数。手动编写的函数可以利用现代 CPU 的 SIMD（单指令多数据）指令，该指令可同时对多个字节执行一次操作，非常适合 UTF-8 文本。bitcannon 将输入的字节视为并行的位流，因此边界是通过整个寄存器的布尔运算得出的，而不是通过逐个字符扫描的方式。