Tokens
Tokens
Introduction Although we interact with LLMs using natural language, these models never process raw text directly. Before a prompt reaches the model, it is converted into a sequence of tokens, the fundamental units that the model understands. Tokenization is one of the earliest stages of the inference pipeline and influences everything from context windows and API pricing to latency and memory usage.
简介 尽管我们使用自然语言与大语言模型（LLM）进行交互，但这些模型从不直接处理原始文本。在提示词（Prompt）到达模型之前，它会被转换为一系列 Token（词元），这是模型所能理解的基本单位。分词（Tokenization）是推理流程中最早的阶段之一，它影响着从上下文窗口、API 定价到延迟和内存使用等方方面面。
What Is a Token? A token is the smallest unit of text processed by a language model; it is not necessarily a word. Depending on the tokenizer, a token may represent: an entire word, part of a word, punctuation, whitespace, numbers, symbols, or emojis. Different models use different tokenizers, so the same text may be split differently depending on the model.
什么是 Token？ Token 是语言模型处理的最小文本单位，它不一定是一个完整的单词。根据分词器的不同，一个 Token 可能代表：一个完整的单词、单词的一部分、标点符号、空格、数字、符号或表情符号。不同的模型使用不同的分词器，因此同一段文本根据模型的不同，其拆分方式也可能不同。
Why Tokens? Simply because language models operate on numbers, not text. Before the transformer can perform any computation, the input must be converted into a numerical representation. The preprocessing pipeline looks like this: Raw Text → Tokenizer → Tokens → Token IDs → Embedding Layer → Embedding Vectors → Transformer The tokenizer splits the input into tokens and each token is then mapped to a unique integer called a token ID, which are passed through the model’s embedding layer, which converts them into dense vectors that become the actual input to the transformer.
为什么需要 Token？ 很简单，因为语言模型处理的是数字而非文本。在 Transformer 执行任何计算之前，输入必须转换为数值表示。预处理流程如下： 原始文本 → 分词器 → Token → Token ID → 嵌入层 → 嵌入向量 → Transformer 分词器将输入拆分为 Token，每个 Token 随后被映射为一个唯一的整数（称为 Token ID）。这些 ID 会通过模型的嵌入层，转换为稠密向量，从而成为 Transformer 的实际输入。
A Real Example
Instead of using hypothetical examples, let’s look at how OpenAI’s tokenizer processes text.
Input: I have no enemies.
OpenAI tokenizes it to: ["I", " have", " no", " enemies", "."]
With the following token IDs: [40, 679, 860, 33974, 13] that have been generated by OpenAI Tokenizer for the “GPT-5.x & O1/3” models. The transformer never sees the original sentence, it only receives the corresponding sequence of token IDs.
实际案例
我们不使用假设性的例子，来看看 OpenAI 的分词器是如何处理文本的。
输入：I have no enemies.
OpenAI 将其分词为：["I", " have", " no", " enemies", "."]
对应的 Token ID 为：[40, 679, 860, 33974, 13]，这是由 OpenAI 分词器为“GPT-5.x & O1/3”模型生成的。Transformer 从未见过原始句子，它只接收相应的 Token ID 序列。
Token IDs After tokenization, every token is replaced with an integer. Conceptually: ” have” → 679 ” no” → 860 ” enemies” → 33974 … The exact numbers differ between models because each tokenizer has its own vocabulary; as these integers are not meaningful by themselves, they simply act as indices into the model’s embedding table.
Token ID 分词后，每个 Token 都会被替换为一个整数。概念上： ” have” → 679 ” no” → 860 ” enemies” → 33974 … 具体的数字在不同模型间会有所不同，因为每个分词器都有自己的词汇表；这些整数本身没有意义，它们只是作为模型嵌入表（Embedding Table）的索引。
From Tokens to Predictions Once converted into embeddings, the transformer begins inference. At each generation step, the model predicts the probability distribution of the next token. The predicted token is appended to the existing sequence, and the process repeats until a stopping condition is reached. Prompt → Tokenizer → Token IDs → Embeddings → Transformer → Predict Next Token → Append Token → Repeat This autoregressive loop is how every modern decoder-only LLM generates text.
从 Token 到预测 一旦转换为嵌入向量，Transformer 就开始推理。在每个生成步骤中，模型都会预测下一个 Token 的概率分布。预测出的 Token 会被附加到现有序列中，该过程不断重复，直到达到停止条件。 提示词 → 分词器 → Token ID → 嵌入向量 → Transformer → 预测下一个 Token → 附加 Token → 重复 这种自回归循环是所有现代仅解码器（Decoder-only）架构的大语言模型生成文本的方式。
Try It Yourself OpenAI Tokenizer: https://platform.openai.com/tokenizer
亲自尝试 OpenAI 分词器：https://platform.openai.com/tokenizer