Should You Be Distilling Your Own Model?
Should You Be Distilling Your Own Model?
你应该蒸馏自己的模型吗？
$87,000 vs roughly $1,000. Those were our first estimates for processing a corpus of 6.5 million semantic atoms with Sonnet or training and running a smaller model ourselves. I was ready to build the smaller model. Our job was narrow. We needed to read financial text and record who said what, what they claimed, and how certain the source was. A wrong speaker or an invented certainty would become a record that other parts of the system trusted. Sonnet was our quality anchor. We would use it to label examples, train a Qwen2.5-7B-Instruct student, and run the student over the corpus. Kepler, my AI data scientist agent, built the training harness. Owning the model felt like an answer to the price problem and, eventually, an advantage we could keep improving.
8.7 万美元对比大约 1000 美元。这是我们最初的估算：前者是使用 Sonnet 处理 650 万个语义原子（semantic atoms）的成本，后者则是我们自己训练并运行一个小模型的成本。我当时准备构建那个小模型。我们的任务很明确：阅读金融文本，记录谁说了什么、他们主张什么，以及来源的确定性如何。错误的发言人或凭空捏造的确定性，会成为系统中其他部分所信任的记录。Sonnet 是我们的质量锚点。我们计划用它来标注样本，训练一个 Qwen2.5-7B-Instruct 学生模型，并在语料库上运行该模型。我的 AI 数据科学家智能体 Kepler 构建了训练框架。拥有模型感觉像是解决了价格问题，并且最终会成为我们可以持续改进的优势。
Then Tycho, our FinOps and science agent, pushed us to test models we could rent directly. The first bake-off used a 23-case, hand-built set of difficult examples:
随后，我们的财务运营（FinOps）与科学智能体 Tycho 推动我们去测试那些可以直接租用的模型。首次对比测试使用了一组由人工构建的、包含 23 个困难案例的测试集：
| Model | Score | Role in the test |
|---|---|---|
| Claude Sonnet | 5 21/23 | Proposed teacher and quality reference |
| DeepSeek V3.2 | 18/23 | Lower-cost API candidate |
| Qwen3-Next-80B-A3B | 8/23 | Open-weight candidate, distinct from our Qwen2.5 student |
| 模型 | 分数 | 测试中的角色 |
|---|---|---|
| Claude Sonnet | 21/23 | 拟定的教师模型及质量参考 |
| DeepSeek V3.2 | 18/23 | 低成本 API 候选模型 |
| Qwen3-Next-80B-A3B | 8/23 | 开源权重候选模型，与我们的 Qwen2.5 学生模型不同 |
At first, that table strengthened the case for distillation. DeepSeek missed more hard cases than Sonnet. A student trained only on Sonnet’s answers would inherit a different problem: the teacher itself missed two cases, so its labels could not teach the correct answer there. We expected to add our own corrections as we found failures.
起初，这张表格加强了进行模型蒸馏的理由。DeepSeek 在困难案例上的失误比 Sonnet 多。仅根据 Sonnet 的答案进行训练的学生模型会继承另一个问题：教师模型本身就漏掉了两个案例，因此它的标签无法在这些地方教授正确的答案。我们预计在发现失败案例时，会加入我们自己的修正。
What the test was missing: The extraction rubric had been treating uncertainty as something to overcome. Sometimes the source does not support a definite speaker or a definite claim. We added “Unknown” as a legitimate output and used the unknown cases to examine where our instructions were pushing a model past the evidence. That gave us a cheaper experiment than another training run: change the rubric, run fresh hard cases, and check that the model had not learned to answer “Unknown” to everything.
测试中缺失的部分：提取准则（rubric）一直将“不确定性”视为需要克服的问题。但有时来源本身并不支持明确的发言人或明确的主张。我们将“未知”（Unknown）添加为合法的输出，并利用这些未知案例来检查我们的指令在何处引导模型超出了证据范围。这比重新进行一次训练实验更便宜：只需修改准则，运行新的困难案例，并检查模型是否学会了对所有问题都回答“未知”。
In a later evaluation, DeepSeek V4.1 Flash with the corrected rubric passed 46/46 cases across three separate held-out sets. The six clear control cases stayed clean. Those 46 cases are separate from the original 23-case bake-off, and V4.1 Flash is a different model from V3.2. The table above and the later result cannot be subtracted to calculate how much improvement came from the rubric alone. The later tests showed that an available API model could clear our bounded hard-case bar under the revised instructions. They did not prove performance across all 6.5 million atoms. We had built a student baseline, but we never benchmarked that student on the hard-case set or served it over the full corpus. There is no honest student score to put beside DeepSeek’s later result.
在后来的评估中，使用修正后准则的 DeepSeek V4.1 Flash 在三个独立的留出集（held-out sets）上通过了全部 46/46 个案例。那 6 个清晰的对照案例也保持准确。这 46 个案例与最初 23 个案例的对比测试是分开的，且 V4.1 Flash 与 V3.2 是不同的模型。上述表格和后来的结果无法通过简单的减法来计算出仅由准则改进带来的提升。后来的测试表明，现有的 API 模型在修订后的指令下可以达到我们设定的困难案例门槛。但它们并没有证明在全部 650 万个原子上的表现。我们虽然构建了一个学生模型基准，但从未在困难案例集上对其进行基准测试，也没有在完整语料库上运行它。因此，没有一个客观的学生模型分数可以与 DeepSeek 后来的结果相提并论。
The bill for the next correction: Our modeled cost for training and running the student was roughly $1,000 in GPU time. The optimized DeepSeek route also modeled at roughly $1,000 for this extraction stage. Neither number is an invoice. The API estimate depends on caching, and the student estimate leaves out the engineering and operations required to serve it. Once the compute estimates were close, the decision moved to something we had barely priced: how long it would take to incorporate the next correction. With an API model, we could revise the rubric and test the next batch. With our student, a correction intended to live in the weights would require new training, evaluation, and deployment. We might still do that someday. For this workload, we had no reason to pay that cycle before testing whether the instructions were the source of the failure.
下一次修正的账单：我们对训练和运行学生模型的建模成本大约是 1000 美元的 GPU 时间。优化后的 DeepSeek 方案在这一提取阶段的建模成本也大约是 1000 美元。这两个数字都不是最终发票。API 的估算取决于缓存，而学生模型的估算则忽略了部署它所需的工程和运维成本。一旦计算成本估算接近，决策就转向了我们几乎没有评估过的因素：纳入下一次修正需要多长时间。使用 API 模型，我们可以修改准则并测试下一批数据。而对于我们的学生模型，如果修正需要写入权重，则需要重新训练、评估和部署。我们将来可能还会这样做，但对于当前的工作负载，在测试指令是否是失败根源之前，我们没有理由支付那样的周期成本。
I had been thinking of the trained model as the asset. It is an asset, but it captures what we knew at training time. The rubric and the test cases are assets too; they can also be wrong or go stale. The advantage I care about now is the rate at which we find an error, inspect the source, correct the judgment, and test that correction against new cases and unaffected controls. We thought we were building that learning into a model we owned. The faster route, at least for this job, was to keep the learning loop outside the weights.
我曾认为训练好的模型才是资产。它确实是资产，但它只捕捉了我们在训练时所知的信息。准则和测试案例也是资产；它们也可能出错或过时。我现在关心的优势是：我们发现错误、检查来源、纠正判断，并针对新案例和未受影响的对照组测试该修正的速度。我们曾以为是将这种学习过程构建进我们拥有的模型中。但至少对于这项工作而言，更快的路径是将学习循环保持在权重之外。
About the cost estimates: The $87,000 was our initial Sonnet estimate; correcting the number of calls reduced the all-Sonnet projection to about $58,000. The student and optimized DeepSeek figures are modeled costs for the extraction stage, not the whole pipeline. The current uncached Fireworks batch projection for DeepSeek is about $2,151. The student was not served over the corpus.
关于成本估算：8.7 万美元是我们最初对 Sonnet 的估算；修正调用次数后，全 Sonnet 方案的预测成本降至约 5.8 万美元。学生模型和优化后的 DeepSeek 数据是提取阶段的建模成本，而非整个流水线。目前 DeepSeek 在 Fireworks 上的未缓存批处理预测成本约为 2151 美元。学生模型并未在整个语料库上运行。