I’m (mostly) picking models on speed now, not intelligence

我现在（主要）是根据速度而非智能来选择模型

For the first time I can remember, I’m not choosing my daily driver models on raw intelligence. I’m choosing them on speed. Have we reached an intelligence tipping point? This is probably going to age like spoilt milk, but right now, models around the ~Opus 4.6 level seem to be ‘smart enough’ for most of my daily tasks - code, pulling together research, designing slide decks and doing analytical tasks against a plethora of databases.

在我记忆中，这是第一次我不再根据原始智能来选择日常使用的模型，而是根据速度。我们是否已经达到了智能的临界点？这番话未来可能会显得很可笑，但就目前而言，Opus 4.6 左右水平的模型似乎已经“足够聪明”，足以胜任我大部分的日常任务——编写代码、汇总研究资料、设计幻灯片以及针对海量数据库进行分析工作。

While like most I was hyped to play around with Fable, ironically the US Gov shutdown gave everyone time to get used to Opus again. When Fable came back post-hype with additional guardrails, the first thing I noticed was just how slow it is. So slow, actually, that I switched back to Opus pretty quickly.

虽然我和大多数人一样，曾对试用 Fable 感到兴奋，但讽刺的是，美国政府的停摆让大家有了重新适应 Opus 的时间。当 Fable 在热度过后带着额外的护栏（guardrails）回归时，我注意到的第一件事就是它太慢了。事实上，它慢到我很快就换回了 Opus。

I’ve spent a lot of my career making software fast. It’s remarkable how much better software feels to interact with when it’s fast. In my experience (and many studies), you can take the most beautiful product, but if it’s slow, you won’t enjoy using it. Equally, you can take a very basic product that’s super fast and it will feel brilliantly utilitarian.

我的职业生涯中花了很多时间在优化软件速度上。令人惊叹的是，当软件运行速度快时，交互体验会好得多。根据我的经验（以及许多研究），即使是最精美的产品，如果运行缓慢，你也不会喜欢使用它。同样，即使是一个非常基础的产品，只要速度极快，也会让人觉得极其好用。

It’s clear to me that when only a few, big, slow models cleared the aforementioned (and hypothetical) intelligence bar, it wasn’t worth the trade off really to use a slower model. Whatever speed you gain you quickly lose in having to redo it because it was broken.

对我来说很清楚，当只有少数几个庞大且缓慢的模型能达到上述（假设的）智能门槛时，使用较慢的模型确实不值得。因为模型出错而不得不重做所浪费的时间，很快就会抵消掉你原本可能获得的任何速度优势。

Is 100tok/s the new 100ms? I’ve written before about agents feeling like dialup, back when frontier models were crawling along at 30-60tok/s. That has changed faster than I expected. The key fact I remember is that to humans, ~100ms feels ‘instant’, the gold standard. I reckon 100tok/s output on a model is about as fast as I can keep up with. After that, it comes in faster than I can (skim) read.

100tok/s（每秒 100 个 token）是新的 100ms 吗？我之前写过，当顶尖模型以 30-60tok/s 的速度缓慢爬行时，智能体（agents）的使用体验就像拨号上网。这种情况的变化速度超出了我的预期。我记得一个关键事实：对人类而言，~100ms 的延迟感觉就是“即时”，这是黄金标准。我认为模型 100tok/s 的输出速度大约是我能跟上的极限。超过这个速度，输出的内容就比我（快速）阅读的速度还要快了。

This isn’t an exact bar, because increasingly most of the model time is spent in reasoning, and not actually showing you output tokens. And also it massively varies on your output, prose gets output with far fewer tokens per character than code, so your mileage may (and will) vary. But roughly, 100-200tok/s to me seems pretty damn fast. Below 50tok/s output feels increasingly slow. Ironically, going past 200tok/s seems almost unnerving - you can try a model out here at 10,000tok/s+ (!). I’m sure this feeling will edge upwards as we get used to it and push our agents to do more complicated work.

这并不是一个精确的界限，因为模型大部分时间越来越多地花在推理上，而不是实际显示输出 token。此外，输出内容的不同也会导致巨大差异：散文的字符 token 比率远低于代码，所以你的实际体验可能会（也一定会）有所不同。但大致来说，对我而言，100-200tok/s 已经非常快了。低于 50tok/s 的输出速度会让人感觉越来越慢。讽刺的是，超过 200tok/s 甚至会让人感到不安——你可以在这里尝试一下 10,000tok/s+ 的模型（！）。我相信随着我们习惯这种速度并推动智能体处理更复杂的工作，这种感觉的阈值会逐渐提高。

Given the plethora of new models that I think are ~clearing the aforementioned bar - such as GLM5.2 and DeepSeek V4 Flash GA - that are open weights and small(er), we now have a wide range of models and speed. If you look at the speed rankings of various providers for GLM5.2 on OpenRouter you can see the enormous range of serving speed - from less than 30tok/s at the bottom to 129tok/s at the top. This is another huge plus to the open weights ecosystem. While there are great benefits in cost that are obvious, the fact that providers are also incentivised to compete on speed like this is really interesting.

鉴于目前有大量我认为已达到上述门槛的新模型——例如 GLM5.2 和 DeepSeek V4 Flash GA——它们不仅是开放权重的，而且体积更小，我们现在拥有了广泛的模型选择和速度表现。如果你查看 OpenRouter 上不同供应商针对 GLM5.2 的速度排名，你会发现服务速度存在巨大差异——从底部的不到 30tok/s 到顶部的 129tok/s。这是开放权重生态系统的另一个巨大优势。虽然成本上的好处显而易见，但供应商也被激励在速度上进行这种竞争，这一点非常有趣。

But there are limitations. If you’re familiar with Pareto’s Principle and Amdahl’s Law you’ll know what’s coming up. Assuming “good enough” models continue to get faster and faster, increasingly the speed benefit is lost to tool calls, and us humans overseeing them. Take an agent using a model processing at 50tok/s. Most of the time is spent waiting for inference to come back. Now run the same turn at 250tok/s and you’ll see that increasingly you are bottlenecked on tool calls on your “local” machine and your decision making. Rough numbers, but the shape holds. The 5x speedup on the model only buys you a 2x speedup on the turn, because the other 25 seconds didn’t move.

但这里也存在局限性。如果你熟悉帕累托法则（Pareto’s Principle）和阿姆达尔定律（Amdahl’s Law），你就会知道接下来会发生什么。假设“足够好”的模型变得越来越快，速度带来的收益会越来越多地被工具调用（tool calls）以及我们人类的监督过程所抵消。以一个使用 50tok/s 模型的智能体为例，大部分时间都花在等待推理结果返回上。现在，如果以 250tok/s 的速度运行同一轮任务，你会发现瓶颈越来越多地出现在你“本地”机器的工具调用和你的决策过程上。虽然数字是粗略的，但逻辑成立：模型 5 倍的速度提升只能为你带来 2 倍的整体任务速度提升，因为剩下的 25 秒时间并没有缩短。

And even worse, making your local machine faster on these tool calls is sort of stalling out, because hardware costs have gone parabolic because of AI. Yet again another weird derivative effect of the AI market. So I suspect (for now at least) there is a limit to how much demand there will be for speed, past a certain point. No doubt there’ll be some examples where huge amounts of reasoning are useful (like mathematics research), and speeding that up is helpful. But I’d expect many agents to start getting bottlenecked on your local/internal hardware, database calls and other bits of latency.

更糟糕的是，提升本地机器处理这些工具调用的速度似乎陷入了停滞，因为 AI 导致硬件成本呈抛物线式增长。这又是 AI 市场带来的一个奇怪的衍生效应。因此，我怀疑（至少目前如此）对速度的需求在超过某个点后是有限的。毫无疑问，在某些领域（如数学研究），大量的推理是有用的，加速这些过程很有帮助。但我预计许多智能体将开始受限于本地/内部硬件、数据库调用以及其他延迟因素。

The price war is coming. Interestingly OpenAI reduced the cost of their Luna variant by 80% just before the DeepSeek V4 Flash GA release, making it remarkably affordable for a frontier model. While I haven’t had as much luck with getting great output out of it vs GLM5.2, I think it points towards an absolute bloodbath of pricing at this end of the market. You can see this happening on OpenRouter with GLM5.2 - endless discounts being offered to try and attract customers in. We’re already down to $0.42/$1.32/MTok on GLM5.2 - 5% of the price of Opus. While the very cheapest is slow, for not much more you can get 109tok/s from DeepInfra.

价格战即将来临。有趣的是，OpenAI 在 DeepSeek V4 Flash GA 发布前夕将其 Luna 变体的成本降低了 80%，使其作为一款前沿模型变得非常实惠。虽然我用它得到的效果不如 GLM5.2，但我认为这预示着该市场领域将迎来一场惨烈的价格战。你可以在 OpenRouter 上看到 GLM5.2 的情况——为了吸引客户，各种折扣层出不穷。GLM5.2 的价格已经降至 $0.42/$1.32/MTok，仅为 Opus 价格的 5%。虽然最便宜的服务速度较慢，但只需多花一点钱，你就能从 DeepInfra 获得 109tok/s 的速度。

As the next generation set of GPUs start being deployed over the next few months - Nvidia’s Vera Rubin series and AMD’s MI400s, amongst others - the new HBM4 memory in those chips will deliver a 2x+ speedup on output tokens from memory bandwidth alone, plus more on top from additional compute and interlink. In 2027 it’s very possible we’ll have very good quality models, at reasonable prices running at 500tok/s+. Staring at “still thinking on xhigh effort” for most of your day may finally become a thing of the past.

随着下一代 GPU（如 Nvidia 的 Vera Rubin 系列和 AMD 的 MI400s 等）在未来几个月内开始部署，这些芯片中新的 HBM4 内存仅凭内存带宽就能带来 2 倍以上的输出 token 速度提升，再加上额外的计算能力和互联性能，速度还会进一步提升。到 2027 年，我们很有可能以合理的价格获得运行速度超过 500tok/s 的高质量模型。整天盯着“正在思考高难度任务”的提示，最终可能成为过去式。

What will be interesting to watch for - and I’m not sure where to bet - is if the vast 2-3T+ param models actually do perform dramatically better for everyday tasks. On one hand it feels like we’ve hit a sweet spot right now, on another having an order of magnitude more intelligence in the model may make that sweet spot look very, very primitive. Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I’m getting deep into a difficult task and I feel I could do with the extra “firepower” that Fable offers, but…

值得关注的是——我还不确定该押注哪一方——那些拥有 2-3T+ 参数的庞大模型在日常任务中是否真的表现得更好。一方面，感觉我们现在已经达到了一个“甜蜜点”；另一方面，拥有高出一个数量级的智能模型可能会让现在的“甜蜜点”看起来非常、非常原始。其次，当然是无休止的护栏触发，它们往往发生在最糟糕的时候——正当我深入处理一项困难任务，觉得需要 Fable 提供的额外“火力”时，但……