How do programming languages impact token efficiency and correctness?

编程语言如何影响 Token 效率与准确性？

This somewhat widely cited post (I keep seeing it cited, anyway) suggests that dynamic languages and/or languages that represent things more concisely are more token efficient. It seems to be cited enough that LLM search results agree. For example, when I searched for “dynamic vs static language token cost” (no quotes), Google’s AI summary opened with Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact. 这篇被广泛引用的文章（无论如何，我一直看到它被引用）指出，动态语言和/或表达更简洁的语言在 Token 使用上更高效。它似乎被引用得足够多，以至于大语言模型（LLM）的搜索结果也认同这一点。例如，当我搜索“动态语言与静态语言的 Token 成本”（不带引号）时，谷歌的 AI 摘要开头便写道：动态类型语言通常比传统的静态类型语言具有更低的 LLM Token 成本，因为省略显式的类型声明使代码更加紧凑。

Google’s AI cited the same post, which suggests that some concise dynamic languages have maybe 1/2 to 1/3 the token cost of static languages like Rust, Go, C++, etc. The author says There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient). And then they later tried J, saying It dominates at just 70 tokens average, nearly half of Clojure (109 tokens). Array languages can be extremely token-efficient when they avoid exotic symbol sets. If token efficiency turns out to be a key driver, this is perhaps a very interesting way for languages to evolve. 谷歌的 AI 引用了同一篇文章，该文章暗示一些简洁的动态语言的 Token 成本可能仅为 Rust、Go、C++ 等静态语言的 1/2 到 1/3。作者表示，C 语言（我比较的语言中 Token 效率最低的）与 Clojure（效率最高的）之间存在 2.6 倍的显著差距。随后他们尝试了 J 语言，称其以平均仅 70 个 Token 的成绩占据主导地位，几乎是 Clojure（109 个 Token）的一半。如果数组语言能避免使用生僻的符号集，它们可以实现极高的 Token 效率。如果 Token 效率被证明是一个关键驱动因素，这或许是编程语言演进的一种非常有趣的方式。

The other dynamic vs. static language token comparison I’ve found floating around is this one, which supports the same conclusion. If you want to treat this as part 8 of this series of exercises on benchmarking, evals, and experimental design, you can click through to the links and think about eval issues before reading further. Without running our own eval, one problem the first experiment has is that the problems are trivial, which we can see from quote above; a problem that can be solved in 70 tokens in J and 109 in Clojure isn’t much of a problem at all (the author used Rosetta Code). 我发现流传的另一个关于动态与静态语言 Token 的比较是这一个，它支持相同的结论。如果你想把这篇内容当作基准测试、评估和实验设计系列练习的第 8 部分，你可以在继续阅读之前点击链接，思考一下评估中存在的问题。在不进行我们自己的评估的情况下，第一个实验存在的一个问题是题目过于简单，从上面的引文中可以看出；一个在 J 语言中只需 70 个 Token、在 Clojure 中只需 109 个 Token 就能解决的问题，根本算不上什么难题（作者使用了 Rosetta Code）。

As we saw when we looked at other evals of caveman mode vs. our own evals, you can get very different results from trivial problems where most of the work is in printing out an answer vs. slightly less trivial problems that actually require some amount of “real work”; the big gains claimed by caveman mode and shown in replications go away when you start looking at problems that take more than just a few tokens. In general, performance on trivial tasks doesn’t generalize. 正如我们在观察“原始人模式”（caveman mode）与其他评估对比时所看到的那样，对于大部分工作仅在于打印答案的简单问题，与那些确实需要一定“实际工作量”的稍复杂问题相比，你会得到截然不同的结果；当开始处理那些不仅仅需要几个 Token 的问题时，原始人模式所声称的巨大收益以及在复现中显示的结果就会消失。总的来说，在简单任务上的表现并不能推广到一般情况。

The issues in the second link are a little more subtle, so we’ll defer most of them to an appendix, but they include issues like one of the tests executing the wrong path (which doesn’t exist), causing a test to fail. One of the later agents then symlinks the non-existent path to its own executable, which works for that case, but also causes every later test to run that one agent’s executable instead of the correct executable. The author tries to draw conclusions about what it means that Rust had some failures, but all it means is that scoring for Rust ran before the Go agent symlinked all scoring on that broken test to the Go executable. 第二个链接中的问题稍微隐蔽一些，所以我们将大部分内容留到附录中，但它们包括诸如其中一个测试执行了错误的路径（该路径不存在），导致测试失败等问题。随后，其中一个代理将该不存在的路径符号链接到它自己的可执行文件，这对于该案例有效，但也导致后续的每个测试都运行了该代理的可执行文件，而不是正确的那个。作者试图就 Rust 出现的一些失败得出结论，但实际上这仅仅意味着 Rust 的评分是在 Go 代理将该损坏测试的所有评分符号链接到 Go 可执行文件之前运行的。

Instead of relying on these evals, we can try running some of our own evals. As we can see from these evals as well as the evals discussed in our last exercises on evals, it’s very easy to make an eval that doesn’t say what the creator of the eval seems to think it’s saying. No doubt these evals will not be an exception to this and will be flawed (see appendix below for more details). As a way to build my intuition about things, I like to pre-register guesses before looking at results. 与其依赖这些评估，我们不如尝试运行我们自己的评估。正如我们从这些评估以及我们在上次评估练习中讨论的评估中所看到的那样，很容易做出一个无法表达其创建者意图的评估。毫无疑问，这些评估也不例外，同样会存在缺陷（详见下文附录）。为了建立对事物的直觉，我喜欢在查看结果之前先预先登记我的猜测。

Some things I pre-registered with friends were: High confidence (95%): the overall dynamic vs. static language claim won’t hold. For reasons stated above: this feels analogous to the caveman eval, where the result will, at best, get diluted as the problem gets larger. Low confidence (60%): static languages will be somewhat better than dynamic at ultra effort. Very weak confidence that, at ultra effort, the harness will get feedback to the model more quickly and this will result in some kind of benefit for either correctness or efficiency, but it would also seem reasonable for this to not be the case for all kinds of reasons. 我与朋友预先登记的一些观点包括：高置信度（95%）：关于动态语言与静态语言的总体结论将无法成立。原因如上所述：这感觉类似于“原始人评估”，随着问题规模的扩大，结果充其量会被稀释。低置信度（60%）：在“超高难度”（ultra effort）下，静态语言会比动态语言稍好一些。极弱的置信度认为，在超高难度下，测试框架能更快地向模型反馈信息，从而在准确性或效率上带来某种好处，但考虑到各种原因，这种情况不成立也是合理的。

e.g., I’ve noticed that codex, when invoking the Rust compiler, very often makes the exact same error and then has to fix it; perhaps this kind of thing dwarfs things like a hypothetical faster feedback cycle. High confidence (98%): the “weird” language supremacy of something like J won’t hold. Same reasoning as the overall static vs. dynamic claim, with the additional thought that AI labs are going to have much less (and possibly zero) synthetic data RL env effort on obscure languages. 例如，我注意到 Codex 在调用 Rust 编译器时，经常犯同样的错误，然后不得不进行修复；也许这类问题的影响远超所谓的“更快的反馈周期”。高置信度（98%）：像 J 语言这类“怪异”语言的优越性将无法成立。理由与动态语言对比静态语言的总体结论相同，此外我还认为，AI 实验室在冷门语言上的合成数据强化学习环境投入会少得多（甚至可能为零）。

Zstd: For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it’s not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn’t intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can “easily” be derived from the RFC that should work. Zstd：在第一次评估中，我尝试给代理提供 zstd RFC（以及勘误表），并要求它们实现一个完整的 zstd 解码器（代理被限制在没有互联网访问的容器中）。测试用例没有提供给代理。对于像 zstd 这样覆盖面广的项目，期望测试覆盖所有可能的情况是不合理的。例如，尽管 zstd 是一个经过充分测试的软件，但我曾经在 zstd 中发现过一个数据损坏的 Bug。测试套件的目的不是为了发现可能潜伏多年的极端边缘情况，而是为了检查可以从 RFC 中“轻松”推导出的、理应正常工作的各种情况。

Below, the x-axis is cost and the y-axis is correctness score (up and to the left is better / down and to the right is worse); average result on medium and ultra efforts with GPT-5.6 Sol. If we only look at medium (and ignore the fact that results often wildly differ on different tasks), we might come to a conclusion like the Alderson evaluation, that dynamic languages are more efficient and better when using LLMs because (ignoring relatively obscure languages) the cluster of dynamic languages lands up and to the left of the cluster of static languages (we used Alderson’s color-coding for static vs. dynamic to make it easy to compare at a glance). But if we look at ultra effort, the results are quite mixed, with a couple static languages doing the best, with more static than dynamic languages among the better results. 下图横轴为成本，纵轴为准确性得分（左上角更好，右下角更差）；这是使用 GPT-5.6 Sol 在中等和超高难度下的平均结果。如果我们只看中等难度（并忽略结果在不同任务上往往差异巨大的事实），我们可能会得出像 Alderson 评估那样的结论，即在使用 LLM 时，动态语言更高效、表现更好，因为（忽略相对冷门的语言）动态语言集群位于静态语言集群的左上方（我们使用了 Alderson 的静态与动态颜色编码，以便一目了然地进行比较）。但如果我们观察超高难度，结果则相当复杂，其中几种静态语言表现最好，且在较好的结果中，静态语言的数量多于动态语言。