How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

通过专门化小型语言模型（SLM）的词汇表，如何在实际应用中将推理成本降低 77%

Models specialized for a domain or a task by fine-tuning general open-source models, such as Qwen and Gemma, inherit a vocabulary built for many domains and general applications. In an essay grader we run in production, that vocabulary split the text the model writes most into many tokens. This article previews some results from the VocabX paper, which we will publish soon: how we specialized the vocabulary around the cost of each token rather than its frequency, and cut measured inference cost by 77% without changing the base model.

通过微调 Qwen 和 Gemma 等通用开源模型而针对特定领域或任务进行专门化的模型，会继承一个为多种领域和通用应用构建的词汇表。在我们生产环境中运行的一款作文评分器中，该词汇表将模型最常书写的文本拆分成了许多标记（token）。本文预览了我们即将发表的 VocabX 论文中的部分成果：我们如何围绕每个标记的成本而非其频率来专门化词汇表，并在不改变基础模型的情况下，将测得的推理成本降低了 77%。

The tokenizer is usually treated as a fixed property of the base model, inherited and left alone. For a model specialized in a single domain, it can be one of the largest cost levers, and it is set before any training step. One of the systems we run in production is an automated grader for Portuguese essays, built on Qwen3-4B. It receives the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency.

分词器通常被视为基础模型的固定属性，被继承后便不再变动。对于一个专门针对单一领域优化的模型而言，分词器可能是最大的成本杠杆之一，且在任何训练步骤开始前就已经设定好了。我们在生产环境中运行的系统之一是基于 Qwen3-4B 构建的葡萄牙语作文自动评分器。它接收作文、提示词和辅助文本，并返回一份包含各项能力评分和评语的 JSON 评估结果。

The model was already small, at 4 billion parameters, and had already been trained for the task. Its tokenizer, however, was still the original Qwen3-4B one, built for many languages and many kinds of text. In the grader, it was used for a single language, Portuguese, and a single response format. That had two costs: the text the grader writes on every request took more tokens than it needed, and at every generated token the model scored a vocabulary of 151,669 entries, including many this task almost never uses.

该模型本身已经很小（40 亿参数），并且已经针对该任务进行了训练。然而，其分词器仍然是原始的 Qwen3-4B 分词器，它是为多种语言和多种文本类型构建的。在评分器中，它仅用于单一语言（葡萄牙语）和单一响应格式。这带来了两项成本：评分器在每次请求中书写的文本占用的标记比实际需要的多；并且在生成每个标记时，模型都要对包含 151,669 个条目的词汇表进行评分，其中许多条目在该任务中几乎从未使用过。

We rebuilt that vocabulary with a method we call VocabX, keeping the same base model, the same data and the same training recipe. The numbers in this article come from the current version of the paper, still under review, and may change before publication.

我们使用一种称为 VocabX 的方法重建了该词汇表，同时保持基础模型、数据和训练方案不变。本文中的数据来自目前尚在审阅中的论文版本，在正式发表前可能会有所变动。

A Vocabulary Built for Everyone

为所有人构建的词汇表

The leading language models read and write in tokens, and the tokenizer decides how text becomes tokens. The tokenizers of the leading general-purpose models are trained on large collections of text from many domains and languages, from source code to contracts to Chinese. For a general-purpose model, that choice makes sense. For a model used in a single domain, it creates two kinds of waste at once.

领先的语言模型以标记的形式进行读写，而分词器决定了文本如何转化为标记。领先的通用模型的分词器是在来自许多领域和语言的大量文本集合上训练的，涵盖了从源代码、合同到中文等各种内容。对于通用模型而言，这种选择是合理的。但对于用于单一领域的模型，它会同时造成两种浪费。

The first is fragmentation. The tokenizer splits into many pieces exactly what the domain repeats most. In the grader, every response starts with the same JSON envelope, the same competency names, the same evaluation phrases. In contracts, it is the boilerplate clauses. In code, the identifiers. Each of these sequences costs several tokens where one would do, and the model pays for every one of them on every request.

第一种是碎片化。分词器将该领域重复最多的内容拆分成了许多碎片。在评分器中，每个响应都以相同的 JSON 外壳、相同的能力名称和相同的评估短语开头。在合同中，这是标准条款；在代码中，这是标识符。这些序列中的每一个都需要多个标记，而本可以用一个标记完成，模型在每次请求中都要为每一个标记付费。

The second is excess. The vocabulary carries tens of thousands of entries the domain never uses, and the model computes a score for all of them at every token it generates. A model tuned to grade American school biology exams will almost never write in Chinese, yet it computes the probability of every Chinese-language token at every step. The vocabulary splits into many pieces what the domain repeats most, and carries entries the domain never uses.

第二种是冗余。词汇表中包含了数以万计该领域从不使用的条目，而模型在生成每个标记时都要为所有这些条目计算分数。一个针对美国学校生物考试评分而调整的模型几乎永远不会用中文书写，但它在每一步都要计算每个中文字符标记的概率。词汇表将该领域重复最多的内容拆分成了许多碎片，并携带了该领域从不使用的条目。

Reading Is Cheap, Writing Is Expensive

阅读廉价，书写昂贵

The natural reaction is to count tokens and try to reduce the count. But the count is not the bill. To see why a generated token costs more, it helps to look at how inference works. It has two phases. In prefill, the model processes the whole input in a single pass through the network: each weight read from GPU memory is reused for every input token, and the limit is compute.

自然的反应是计算标记数量并尝试减少它。但标记数量并不等同于账单金额。要了解为什么生成的标记成本更高，有必要看看推理是如何工作的。它有两个阶段。在预填充（prefill）阶段，模型通过网络一次性处理整个输入：从 GPU 内存读取的每个权重都会被每个输入标记重复使用，此时的瓶颈是计算能力。

In decode, the model generates the response one token at a time, and each token needs a full pass. Each weight read is used only a few times, and the GPU spends most of its time waiting for data to arrive from memory: the limit becomes memory bandwidth. This effect is strongest exactly where we operate: small models served with moderate batches. Batching requests splits the weight reads across them and eases the problem, but does not solve it.

在解码（decode）阶段，模型一次生成一个标记，每个标记都需要进行一次完整的传递。每次读取的权重只被使用几次，GPU 大部分时间都在等待数据从内存中到达：此时的瓶颈变成了内存带宽。这种效应在我们所处的领域最为明显：即使用中等批次服务的小型模型。批处理请求可以在它们之间分摊权重读取并缓解该问题，但无法彻底解决它。

At every decode step, the model also reads the KV cache, the attention memory that holds the context already processed and grows with it. In the case we measured, the grader, a generated token costs 7.4 times a token read, but the ratio varies with the hardware, the problem, the batch size, the context length and other factors, and it can be larger than that. In general, the direction holds: generating costs more than reading.

在每个解码步骤中，模型还会读取 KV 缓存，即保存已处理上下文并随之增长的注意力内存。在我们测量的评分器案例中，生成一个标记的成本是读取一个标记的 7.4 倍，但该比例会随硬件、问题、批次大小、上下文长度和其他因素而变化，甚至可能更高。总的来说，这一方向是成立的：生成比读取更昂贵。

Think of a shipping company that charges per box, but charges seven times more for outgoing boxes than for incoming ones. Reducing the total number of boxes helps. Reducing the outgoing boxes helps seven times more. A method that compresses input and output equally misses most of the possible savings.

想象一家按箱收费的航运公司，但对出库箱的收费是入库箱的七倍。减少箱子总数是有帮助的，但减少出库箱的帮助要大七倍。一种同等压缩输入和输出的方法会错失大部分潜在的节省空间。

The third component of the bill is the vocabulary itself. The model’s last layer, the LM head, turns the internal state into a score for every possible token. To do that, it holds one row of parameters per vocabulary entry. In a model the size of Qwen3-4B, that matrix accounts for almost 10% of the parameters. It is a share that large models absorb without noticing, but it weighs on a small model.

账单的第三个组成部分是词汇表本身。模型的最后一层（LM head）将内部状态转化为每个可能标记的分数。为此，它为每个词汇表条目保留了一行参数。在 Qwen3-4B 这种规模的模型中，该矩阵占了近 10% 的参数。对于大型模型来说，这个比例可以忽略不计，但对于小型模型来说，这却是一个沉重的负担。

And the cost is not paid only once. The input layer looks up only the row of the token being read, but the LM head reads and multiplies the entire matrix for every generated token. Since decode is already memory-bound, every extra row is more data to move at every step. An entry the domain never uses costs exactly the same as a frequent one. That is why adding tokens can lower the count and raise the bill at the same time. VocabX does not include this term in its selection criterion, but it controls vocabula

而且这种成本并非只支付一次。输入层仅查找正在读取的标记所在的那一行，但 LM head 在生成每个标记时都要读取并乘整个矩阵。由于解码阶段已经受限于内存，每一行额外的参数都意味着每一步需要移动更多的数据。一个该领域从不使用的条目与一个频繁使用的条目成本完全相同。这就是为什么增加标记有时会降低标记数量，却同时提高了账单金额。VocabX 在其选择标准中并未包含这一项，但它控制了词汇表。