Kimi K3, and what we can still learn from the pelican benchmark

Kimi K3，以及我们能从“鹈鹕基准测试”中学到什么

Chinese AI lab Moonshot AI announced Kimi K3 this morning, describing it as their “most capable model to date, with 2.8 trillion parameters”. It’s currently available via their website and API, but an open weight release is promised “by July 27, 2026”. 中国人工智能实验室月之暗面（Moonshot AI）于今日上午发布了 Kimi K3，称其为“迄今为止能力最强的模型，拥有 2.8 万亿参数”。该模型目前已通过其网站和 API 提供服务，并承诺将于“2026 年 7 月 27 日前”发布开源权重版本。

Moonshot are calling this the first “open 3T-class model” (I guess they’re rounding 2.8 trillion up to 3 trillion), taking the crown from DeepSeek’s 1.6T v4 Pro. Their self-reported benchmarks have K3 mostly beating Claude Opus 4.8 max and GPT-5.5 high, while losing out to Claude Fable 5 and GPT-5.6 Sol. Moonshot 将其称为首个“开源 3T 级模型”（我猜他们是将 2.8 万亿四舍五入到了 3 万亿），从 DeepSeek 的 1.6T v4 Pro 手中夺过了桂冠。根据他们自测的基准数据，K3 在大多数指标上超越了 Claude Opus 4.8 max 和 GPT-5.5 high，但略逊于 Claude Fable 5 和 GPT-5.6 Sol。

A few highlights from the Artificial Analysis report on the model: 以下是 Artificial Analysis 关于该模型的报告中的几个要点：

• “On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5.” “在我们私有的长周期知识工作评估中，Kimi K3 的总 Elo 分数为 1547，较 Kimi K2.6 提升了 732 分，仅次于 Claude Fable 5。”
• “Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers.” “单任务成本（0.94 美元）与 GPT-5.6 Sol（1.04 美元）相当，约为 Opus 4.8（1.80 美元）价格的一半，但高于开源权重同类模型。”
• “Kimi K3’s token usage on the Artificial Analysis Intelligence Index decreased significantly, using 21% fewer output tokens than K2.6.” “Kimi K3 在 Artificial Analysis 智能指数上的 Token 使用量显著下降，输出 Token 比 K2.6 少了 21%。”

The model is also now the leading model on Arena.ai’s Frontend Code arena, surpassing even Claude Fable 5. The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic’s Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date. This is a significant increase on their earlier models such as Kimi K2.6 at $0.95/$4. 2.8 trillion parameters is also more than twice the size of that 1T model. 该模型目前也是 Arena.ai 前端代码竞技场中的领先模型，甚至超过了 Claude Fable 5。这款新模型的定价值得注意：输入 Token 每百万 3 美元，输出 Token 每百万 15 美元，这使其与 Anthropic 的 Claude Sonnet 系列处于同一水平，并成为中国 AI 实验室迄今为止发布的最昂贵的模型。相比他们早期的模型（如 Kimi K2.6 的 0.95 美元/4 美元），价格有了显著上涨。2.8 万亿参数也超过了之前 1T 模型规模的两倍。

But how does it pelican? I used OpenRouter (to avoid signing up for a Moonshot API key) with the llm-openrouter plugin to generate an SVG of a pelican riding a bicycle. 那么，它的“鹈鹕测试”表现如何？我使用了 OpenRouter（为了避免注册 Moonshot API 密钥）配合 llm-openrouter 插件，生成了一张鹈鹕骑自行车的 SVG 图片。

What can we learn from the pelican? My “Generate an SVG of a pelican riding a bicycle” test is 21 months old now. It was never a particularly great benchmark. It started out as a joke on how absurdly difficult it is to compare these models, but then for the first year it turned out to have a surprising correlation to how good the models actually were. That connection has been mostly severed now. 我们能从鹈鹕中学到什么？我的“生成一张鹈鹕骑自行车的 SVG”测试已经有 21 个月了。它从来都不是一个特别好的基准测试。最初，它只是为了调侃比较这些模型有多么荒谬的困难，但在第一年里，它竟然与模型的实际表现有着惊人的相关性。现在，这种联系已经基本断裂了。

The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length. So don’t go using pelicans to compare models! 鹈鹕测试最大的局限性在于，它完全没有触及当今模型最重要的能力：智能体工具调用，以及在对话长度增加时可靠地操作工具的能力。所以，别再用鹈鹕来比较模型了！

All of that said, I still get a decent amount of value out of running the benchmark myself. Firstly, it’s a forcing function for actually trying the model. If I show you a pelican, that means I’ve managed to run a prompt through it. If the model has an official API I’ll use that, if it’s open weight (and small enough to fit a 128GB M5 MacBook Pro) I’ll try running it on my own machine, usually via llama.cpp or LM Studio or Ollama. 话虽如此，我自己运行这个基准测试仍然能获得不少价值。首先，这是一种强制手段，让我真正去尝试使用模型。如果我向你展示了一只鹈鹕，那就意味着我已经成功运行了一个提示词。如果模型有官方 API，我会使用它；如果是开源权重（且足够小，能装进 128GB 内存的 M5 MacBook Pro），我会尝试在自己的机器上运行，通常通过 llama.cpp、LM Studio 或 Ollama。

More importantly though, even the act of a single prompt to “Generate an SVG of a pelican riding a bicycle” can reveal interesting model characteristics. Consider the result for Kimi K3 today. Running those simple prompts helped emphasize several points about the model. It only has one reasoning effort right now, “max”—and it shows. The model consumed 13,241 reasoning tokens to output 3,417 tokens of response. This is expensive—the pelican cost 25 cents! 更重要的是，仅仅是“生成一张鹈鹕骑自行车的 SVG”这一个提示词，就能揭示模型有趣的特性。以今天 Kimi K3 的结果为例，运行这些简单的提示词有助于强调关于该模型的几个要点。它目前只有一种推理强度，即“max”——而且表现得很明显。该模型消耗了 13,241 个推理 Token 才输出了 3,417 个响应 Token。这很昂贵——这只鹈鹕花了 25 美分！

Really though the main things I gain from the pelican test are: 实际上，我从鹈鹕测试中获得的主要收获是：

1. It’s a “hello world” exercise for prompting a model. 它是提示词工程的“Hello World”练习。
2. A rough cost and reasoning estimate for a simple task. 对简单任务的成本和推理能力的粗略评估。
3. Confirmation that the model can output valid SVG and has a basic idea of geometry and spatial awareness. 确认模型能够输出有效的 SVG，并具备基本的几何和空间感知概念。