Better AI code comment detector
Better AI code comment detector
更优秀的 AI 代码注释检测器
When I trained the previous ai comment classifier, I used partially personal private data to do it, and built it on a somewhat shaky foundation, so I couldn’t share the code or data. I rebuilt it on public data and a better foundation! 在我训练上一个 AI 注释分类器时，由于使用了部分个人隐私数据，且构建基础不够稳固，因此我无法分享代码或数据。现在，我基于公开数据和更坚实的基础对其进行了重构！
First off, you might want to try it out. Nothing you paste into that web page leaves your browser, so you can safely try it with whatever you like. I have invited some testers to try out an earlier version of it, and they had mainly positive feedback to give. 首先，你可能想亲自试用一下。你粘贴到网页中的任何内容都不会离开你的浏览器，所以你可以放心地用任何内容进行测试。我曾邀请了一些测试人员试用早期版本，他们给出的反馈主要是积极的。
We won’t break down robot-isms the way we broke down Claude-isms in the previous article, because in the ui of the new classifier you can just click any part of the text being classified to see which features activate on that portion of the text, and how they contribute to the overall judgment. Here’s an example of the expanded feature activation view. 我们不会像上一篇文章分析“Claude 风格”那样去拆解“机器人风格”，因为在新分类器的界面中，你只需点击被分类文本的任何部分，就能看到哪些特征在该部分被激活，以及它们如何影响最终的判断结果。以下是展开后的特征激活视图示例。
In terms of performance, the headline number is the balanced accuracy of 77 %. This is how often the classifier gets the human vs. robot verdict right, assuming human-written and robot-generated comments are equally likely. The classifier also prints a predicted percentage which is calibrated, meaning it can be read as the probability that any specific verdict is correct. 在性能方面，核心指标是 77% 的平衡准确率。假设人类编写的注释和机器人生成的注释概率相等，这就是分类器在区分人类与机器人结论时的正确率。分类器还会输出一个经过校准的预测百分比，这意味着它可以被解读为特定结论正确的概率。
We test this through the calibration curve, which shows what probability the classifier assigns to an event with a known probability. Since all dots lie very close to the reference diagonal, we know they are approximately correct. This holds true across comments of multiple lengths, where a fitted temperature parameter adjusts for increased confidence as the amount of data increases. 我们通过校准曲线来测试这一点，该曲线显示了分类器为已知概率的事件分配了多少概率。由于所有点都非常接近参考对角线，我们知道它们大致是准确的。这一点在不同长度的注释中同样适用，其中拟合的温度参数会随着数据量的增加而调整，以提高置信度。
We can get more details about the classifier’s failure modes by looking at its confusion matrix. In this table, “robot” is considered the positive class, i.e. the thing we want to detect. The abbreviations stand for true/false positive/negative rate. 通过查看混淆矩阵，我们可以更详细地了解分类器的失效模式。在此表中，“机器人”被视为正类，即我们想要检测的目标。缩写分别代表真/假阳性率和真/假阴性率。
| verdict: human | verdict: robot |
|---|---|
| input: human | tnr = 0.73 |
| input: robot | fnr = 0.20 |
When presented with a known-human input, the classifier correctly judges it as human 73 % of the time. With a known-robot input, it is correctly judged 80 % of the time. This means in both cases (known-human and known-robot) the mistake rate is around 25 %. That might sound high! But remember that this mistake rate is the aggregate over all possible inputs. 当输入已知为人类编写的内容时，分类器有 73% 的概率正确判断为人类。当输入已知为机器人生成的内容时，正确判断率为 80%。这意味着在这两种情况下（已知人类和已知机器人），错误率都在 25% 左右。这听起来可能很高！但请记住，这个错误率是所有可能输入的汇总结果。
We don’t need to pay too much attention to it, because the classifier outputs a calibrated predictive percentage every time it classifies something. Thus, for individual judgments, we know when the risk of false positives is lower or higher. When the classifier is very confident – e.g. when the confidence is 80 % or more – the risk of a false positive drops to 5 %. When the classifier is uncertain – when confidence is around 50 % – then by calibration it will issue the wrong verdict around half the time. 我们不必过于关注这一点，因为分类器每次分类时都会输出一个经过校准的预测百分比。因此，对于单次判断，我们知道假阳性风险何时较低或较高。当分类器非常有信心时（例如置信度在 80% 或以上），假阳性风险会降至 5%。当分类器不确定时（置信度在 50% 左右），根据校准，它大约有一半的时间会给出错误的结论。
I mention the numbers in this confusion matrix only because they are so often used when discussing classifiers, so more academically inclined readers may expect to see it. Here are some other requested numbers: 我提到混淆矩阵中的这些数字，仅仅是因为它们在讨论分类器时经常被用到，学术倾向较强的读者可能希望看到这些数据。以下是其他一些被要求的指标：
- Accuracy: 77 %
- Precision: 75 %
- Recall: 80 %
- Sensitivity: 80 %
- Specificity: 73 %
- F1 score: 77 %
The accuracy, precision, and F1 score depend on the base rate, but here they are computed from an ignorance assumption, i.e. an equal mix of human-written and robot-generated comments. All of these numbers come from cross-validation. I have also manually tested a smaller non-synthetic set of real-world comments from humans and robots to see how well the classifier generalises slightly out of sample. 准确率、精确率和 F1 分数取决于基准率，但此处它们是基于“无知假设”计算的，即人类编写和机器人生成的注释各占一半。所有这些数字均来自交叉验证。我还手动测试了一组较小的、非合成的真实世界人类和机器人注释，以观察分类器在样本外数据的泛化能力。
| verdict: human | verdict: robot |
|---|---|
| input: human | tnr = 0.89 |
| input: robot | fnr = 0.14 |
This translates to the following performance numbers: 这转化为以下性能指标：
- Accuracy: 88 %
- Precision: 89 %
- Recall: 86 %
- Sensitivity: 86 %
- Specificity: 89 %
- F1 score: 87 %
This is very good! It looks like non-synthetic, more real-worldy cases are easier for the classifier to discriminate between than the training data. Of course, all of this is tested with code comments only. The classifier is not built to detect robot-generated texts of other kinds. It can do it, but I make no promises of its accuracy. With that out of the way, let’s talk about how it’s made. 这非常棒！看起来非合成的、更接近真实世界的案例比训练数据更容易让分类器进行区分。当然，所有这些测试仅针对代码注释。该分类器并非为检测其他类型的机器人生成文本而构建。它或许能做到，但我无法保证其准确性。言归正传，让我们谈谈它是如何制作的。
Data collection
数据收集
The first step, as before, is to build a good data set. Ideally, we’d plan this meticulously and do it right the first time. If we do that, it should cost us about $30 to get the dataset that powers this classifier. It contains enough data to reach diminishing returns in discriminating between the more similar models. It is possible to extract a more powerful classifier with more data, but it would start to be very expensive since classifier power appears to scale with the log of money spent. 第一步，和之前一样，是构建一个好的数据集。理想情况下，我们会精心规划并一次性做好。如果这样做，获取驱动该分类器的数据集成本大约只需 30 美元。它包含足够的数据，在区分更相似的模型时已达到边际收益递减点。通过更多数据提取出更强大的分类器是可能的，但这会变得非常昂贵，因为分类器的能力似乎与投入资金的对数成正比。
That is, if you plan it out and do it right the first time. I didn’t do that. I discovered much later, when evaluating features, that the data I had was junk and I had to collect it all over. Then after a while I discovered again that the data was still junk and had to be recollected again. 前提是你规划好并一次性做对。我没有做到。我在评估特征时才发现数据是垃圾，不得不重新收集。过了一段时间，我又发现数据依然是垃圾，不得不再次重新收集。
The general idea was to find a set of permissively licenced or copy-left repositories, check out their latest commit from the year 2021, and then take a few random files from that commit. These contain human comments. Then we strip out all comments from those files, and have llms generate new comments for the same files. That provides us with robot comments. 总体的思路是：找到一组采用宽松许可或 Copyleft 协议的仓库，检出它们 2021 年的最新提交，然后从中随机抽取几个文件。这些文件包含人类编写的注释。接着，我们删除这些文件中的所有注释，并让大语言模型（LLM）为相同的文件生成新注释。这就为我们提供了机器人注释。
As long as we try to keep the number tokens for each file balanced between all classes (humans and llm models), we can avoid subject matter leakage, where the classifier learns to distinguish files or repositories rather than the style of the text itself. The general idea is simple! But the devil’s where the devil usually is. Here are some mistakes I made, in no particular order: 只要我们努力保持每个文件在所有类别（人类和 LLM 模型）之间的 Token 数量平衡，就能避免“主题泄露”，即分类器学会了区分文件或仓库，而不是文本本身的风格。总体思路很简单！但魔鬼往往藏在细节中。以下是我犯的一些错误（排名不分先后）：
- Accidentally picking different source files for each llm to generate comments for. This causes subject matter leakage.
- 无意中为每个 LLM 生成注释选择了不同的源文件。这导致了主题泄露。
- Generating llm comments for files with very few human comments. This also causes subject matter leakage over the human–robot barrier.
- 为人类注释极少的文件生成 LLM 注释。这也导致了跨越人类-机器人界限的主题泄露。