GigaToken: ~1000x faster Language model tokenization

GigaToken：语言模型分词速度提升约 1000 倍

Gigatoken ~1000x faster than HuggingFace’s tokenizers, drop-in replacement. Tokenize your text data at GB/s! Note that both HF tokenizers and tiktoken are already running multithreaded Rust! Gigatoken 比 HuggingFace 的分词器快约 1000 倍，且可直接替换使用。它能以 GB/s 的速度处理文本数据！请注意，HF 分词器和 tiktoken 本身已经在使用多线程 Rust 运行了！

What is Gigatoken? Gigatoken is the fastest tokenizer for language modeling. It supports a wide range of CPU hardware, and nearly all commonly used tokenizers. See the Benchmarks section for detailed throughput numbers across tokenizers and CPUs. 什么是 Gigatoken？Gigatoken 是目前用于语言模型的最快分词器。它支持多种 CPU 硬件，并兼容几乎所有常用的分词器。有关不同分词器和 CPU 的详细吞吐量数据，请参阅“基准测试”部分。

Installation

安装

pip install gigatoken

Usage

使用方法

Gigatoken can be used with its own API, or in compatibility mode with HuggingFace Tokenizers or Tiktoken. Gigatoken 既可以使用其自带的 API，也可以通过兼容模式与 HuggingFace Tokenizers 或 Tiktoken 配合使用。

Compatibility Mode (Easiest) 兼容模式（最简单）

import gigatoken as gt

# Minimum change from existing HuggingFace tokenizers usage (compatibility mode)
# 与现有 HuggingFace 分词器用法相比改动极小（兼容模式）
hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()

# tokenizer can be used in the same contexts as hf_tokenizer
# tokenizer 可在与 hf_tokenizer 相同的上下文中使用
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

# OR with tiktoken
# 或者使用 tiktoken
tiktokenizer = ...
tokenizer = gt.Tokenizer(tiktokenizer).as_tiktoken()

# Now works like existing tiktoken tokenizers
# 现在它的工作方式与现有的 tiktoken 分词器一样
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance. You can still expect way faster performance across the board, but not quite the 1000x you will get with the Gigatoken API. 为了确保在此设置下输出结果与 HuggingFace Tokenizers 完全一致，我们投入了大量精力，但这会带来不可忽视的性能损耗。虽然你仍然可以获得比以往快得多的性能，但无法达到使用 Gigatoken 原生 API 时那种 1000 倍的提升。

Gigatoken API (Fastest) Gigatoken API（最快）

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B") # Accepts HF model names / 接受 HF 模型名称
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"<|endoftext|>")
tokens = tokenizer.encode_files(file_source)

Using the Gigatoken API lets the Rust implementation read data directly, and skips as much overhead as possible while allowing for maximum parallelism. Keep in mind that passing Python data structures through this API still incurs the overhead of reading from Python. 使用 Gigatoken API 可以让 Rust 实现直接读取数据，在尽可能跳过开销的同时实现最大程度的并行化。请记住，通过此 API 传递 Python 数据结构仍然会产生从 Python 读取数据的开销。

Benchmarks

基准测试

Note: Due to space constraints, the full benchmark tables for AMD EPYC, Apple M4 Max, and AMD Ryzen 7 have been summarized. Please refer to the original repository for the complete data. 注：由于篇幅限制，AMD EPYC、Apple M4 Max 和 AMD Ryzen 7 的完整基准测试表格已进行摘要处理。请查阅原始仓库以获取完整数据。

Benchmark details

基准测试详情

OWT (openwebtext) was chosen because it’s roughly representative of the text you get after extraction from CommonCrawl documents. Gigatoken encodes the whole file un-split, and is thus doing more work than the other tokenizers to find the split boundaries and automatically parallelize. HuggingFace tokenizers (encode_batch_fast) gets the first 100 MB and tiktoken (encode_ordinary_batch) the first 1 GB, both presplit on <|endoftext|>. This is fair because neither of the compared tokenizers do caching, meaning the speed is roughly uniform throughout processing. Tiktoken rows are currently only filled in for tokenizers with official support. The slowest rows are the SentencePiece-based tokenizers, which are not well optimized. 选择 OWT (openwebtext) 是因为它大致代表了从 CommonCrawl 文档中提取后的文本。Gigatoken 对整个文件进行未拆分编码，因此在寻找拆分边界和自动并行化方面比其他分词器做了更多工作。HuggingFace 分词器 (encode_batch_fast) 处理前 100 MB，tiktoken (encode_ordinary_batch) 处理前 1 GB，两者均预先按 <|endoftext|> 进行了拆分。这是公平的，因为被比较的分词器都不进行缓存，这意味着处理过程中的速度大致均匀。Tiktoken 行目前仅针对有官方支持的分词器进行了填充。最慢的行是基于 SentencePiece 的分词器，它们并未得到很好的优化。