Measuring the sloppiness of code
Measuring the sloppiness of code
衡量代码的“邋遢”程度
LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. 大语言模型（LLMs）在生成代码方面已经近乎完美，但这并不是故事的终点。代码在形式上正确，并不意味着它没有引入不必要的抽象、制造重复内容，或者在整体上做出糟糕的决策。
This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC). This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up. 这并非什么惊人的发现。大多数有过“凭感觉编程”（vibe-coded）项目经验的人都意识到，每增加一个新功能，有时会导致代码行数（LOC）呈爆炸式增长。这导致人类失去了对代码的掌控力，因为在那些每月增加数百万行代码的项目中，人类很难跟上进度。
Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can’t really deal with the slop either. 有些人可能会说这根本不是问题，因为他们相信自己的 AI 智能体能处理好。但我有个坏消息要告诉你：智能体其实也无法处理这些“邋遢”的代码（slop）。
Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing. 我拥有物理学背景，在解决问题时总是倾向于采用实验性或定量的方法。当我加入 Earendil 并负责研究如何衡量代码的“邋遢”程度时，我的本能反应是先深入研究相关文献，看看其他公司在做什么。
To be frank, with the exception of a few insightful research papers, I was disappointed at how “vibes based” the industry seems at the moment. In my research and on X, I was constantly bombarded with messages such as “End-to-end coding agents”, “AI that doesn’t just suggest code—it ships it” or “Human-level evaluation without human-level cost”. 坦白说，除了少数几篇有见地的研究论文外，我对目前行业内那种“凭感觉”的现状感到失望。在我的研究过程中以及在 X（原 Twitter）上，我不断被各种信息轰炸，比如“端到端编程智能体”、“不仅能建议代码，还能直接交付的 AI”，或者“无需人类成本的人类水平评估”。
Which like all good tales, have a grain of truth in them. LLMs are able to write almost perfectly correct code. This is because of the scalability and the verifiability of code. It is pretty straightforward to let LLMs generate code and then let that code be checked by hidden tests, which results in a clear reward signal. 这些说法就像所有精彩的故事一样，包含了一丝真相。LLMs 确实能够写出近乎完美正确的代码，这得益于代码的可扩展性和可验证性。让 LLMs 生成代码，然后通过隐藏测试来检查这些代码，从而获得明确的奖励信号，这非常直接。
In stark contrast to that, checking the ‘sloppiness’ of this code often requires human intuition and taste, and is an extremely difficult task in general. I think the best way to illustrate why that is, is by going through possible ways of measuring slop. 与此形成鲜明对比的是，检查代码的“邋遢”程度通常需要人类的直觉和品味，这在总体上是一项极其困难的任务。我认为说明这一点的最好方法，是梳理一下衡量“邋遢”程度的几种可能方式。
AI as a judge: This is probably the most common way of evaluating code quality in the industry and from my observations it rarely works. The most naive way of doing it, namely asking the models how good the code is on a scale from 1-10, is basically equivalent to a random number generator. AI 作为裁判： 这可能是目前行业内评估代码质量最常见的方法，但根据我的观察，它很少奏效。最幼稚的做法是让模型给代码打分（1-10分），这基本上等同于随机数生成器。
The more sophisticated approach, namely trying to give the judge model two solutions A and B, and then letting it decide which solution it prefers, has the downside of the model changing its preference, when you rename the solutions. I am being a bit facetious here and the effect isn’t as pronounced with larger models, but the main point still stands. 更复杂的方法是给裁判模型提供两个方案 A 和 B，让它决定更倾向于哪一个。但这种方法的缺点是，当你重命名这些方案时，模型的偏好可能会改变。我这里说得有点夸张，在大模型中这种效应没那么明显，但核心观点依然成立。
Asking LLMs to judge the code they write is not a substitute for a proper evaluation. Even though there are some interesting approaches with rubrics or the LLMs writing tests, they are still a far shot from actually getting rid of the slop. 让 LLMs 评价自己写的代码并不能替代正规的评估。尽管有一些有趣的尝试，比如使用评分标准或让 LLMs 编写测试，但它们距离真正消除“邋遢”代码还差得很远。
Human judges the AI: If we ignore the fact that there is huge diversity in the quality of software-engineers, this would be the best solution to assure that the code stays human readable. With the downside being that this is not scalable for training AI or having large benchmarks with multiple model providers and harnesses. 人类作为裁判： 如果我们忽略软件工程师水平参差不齐这一事实，这确实是确保代码保持人类可读性的最佳方案。但其缺点在于，对于训练 AI 或构建包含多个模型提供商和测试框架的大型基准测试来说，这种方法无法扩展。
The simplest method: In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure. 最简单的方法： 在我的研究和测试中，仅仅统计代码行数（LOC）的变化，竟成了衡量“邋遢”程度的一种出奇有效的指标。讽刺的是，如果我们开始针对这一指标进行优化，它就不再是一个有意义的衡量标准了。
The next two measures were introduced to me by the paper SlopCodeBench, and seemed promising because they were able to separate legacy code bases from LLM-slop quite well. 接下来的两种衡量指标来自《SlopCodeBench》这篇论文，它们看起来很有前景，因为它们能够很好地将遗留代码库与 LLM 产生的“邋遢”代码区分开来。
Verbosity: Tries to measure the amount of duplicated and unnecessary verbose lines. 冗余度（Verbosity）： 试图衡量重复和不必要的冗余代码行数。
Erosion: Tries to measure how much of a codebase’s mass is concentrated in a few large and complex functions. 侵蚀度（Erosion）： 试图衡量代码库中有多少“质量”集中在少数几个庞大且复杂的函数中。
If we look at the average verbosity and erosion of the code generated during the SlopCodeBench evaluation and compare that to a set of established repos there is a stark difference between them. On average the verbosity in the repos is 0.15 ± 0.06 and in the agents code is 0.33 ± 0.10. For erosion the repos achieve 0.31 ± 0.17 and the agents 0.68 ± 0.20. The agent’s code is on average roughly twice as verbose and eroded as human code. 如果我们观察 SlopCodeBench 评估中生成的代码的平均冗余度和侵蚀度，并将其与一组成熟的代码库进行比较，会发现两者之间存在巨大差异。成熟代码库的平均冗余度为 0.15 ± 0.06，而智能体生成的代码为 0.33 ± 0.10。在侵蚀度方面，成熟代码库为 0.31 ± 0.17，而智能体为 0.68 ± 0.20。智能体生成的代码平均比人类编写的代码冗余度和侵蚀度高出约两倍。
I then investigated some vibe coded projects of my own and a lot of them had a verbosity of up to 0.4 and erosion as high as 0.75, so these results probably weren’t just an artifact of the evaluation. 随后，我调查了自己的一些“凭感觉编程”项目，发现其中许多项目的冗余度高达 0.4，侵蚀度高达 0.75，因此这些结果很可能不仅仅是评估过程产生的偏差。
To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions… 回到为什么智能体无法（真正）自行处理这些“邋遢”代码的问题，我们需要看看 SlopCodeBench 的评估方式。与其他在开始时给智能体提供完整指令列表并设置隐藏测试的基准测试不同，它们采取了相反的做法。它们创建了多轮指令和测试迭代，并在检查点之间清除模型的上下文。这更贴近人类实际使用编程智能体的迭代过程。其结果是，糟糕的编码决策……