Prompt Injection Attacks Are Thwarting AI Hacking Agents

提示词注入攻击正在挫败 AI 黑客代理

Prompt injections, the malicious commands attackers embed into content to entice large language models to follow them, have been attackers’ go-to tool for turning AI platforms against their users. A well-phrased command sneaked into an email or calendar invitation is often all it takes to cause the LLM to exfiltrate sensitive data or follow other harmful actions.

提示词注入（Prompt injections）是指攻击者嵌入内容中的恶意指令，旨在诱导大语言模型（LLM）执行这些指令。这已成为攻击者将 AI 平台转而用于攻击用户的首选工具。只需在电子邮件或日历邀请中潜入一条措辞巧妙的指令，往往就足以导致 LLM 泄露敏感数据或执行其他有害操作。

Now, defenders are embracing the prompt injection, too. Researchers from Tracebit on Monday said they found that placing prompt injections alongside passwords, cryptographic keys, and other secrets stored on Amazon Web Services was often all that was needed to shut down attacks from AI hacking agents. The prompts direct the attacking LLM to perform an action forbidden by its guardrails, the safety barriers AI developers erect to prevent it from taking harmful actions. The LLM responds by shutting down.

现在，防御者也开始拥抱提示词注入技术。Tracebit 的研究人员周一表示，他们发现，将提示词注入与存储在亚马逊云科技（AWS）上的密码、加密密钥及其他机密信息放在一起，往往足以阻止 AI 黑客代理的攻击。这些提示词会引导攻击性 LLM 执行其“护栏”（即 AI 开发人员为防止模型采取有害行动而设置的安全屏障）所禁止的操作，从而导致 LLM 触发防御机制并停止运行。

Examples are a prompt that orders the LLM to provide steps for developing inhalable Anthrax spores, or, in the case of LLMs from Chinese developers, make references to the iconic Tank Man from the 1989 Tiananmen Square massacre. Once the LLM encounters these forbidden commands, it no longer follows its existing commands. The researchers have named the technique context bombing.

例如，一条指令可以要求 LLM 提供开发吸入式炭疽孢子的步骤；或者，对于中国开发者的 LLM，则要求其提及 1989 年天安门广场事件中的标志性人物“坦克人”。一旦 LLM 遇到这些被禁止的指令，它就不会再执行其原有的任务。研究人员将这种技术命名为“上下文轰炸”（context bombing）。

“Ultimately we’re triggering a refusal mechanism in the context,” said Andy Smith, cofounder and CEO of Tracebit, when explaining the name choice. “What we’re trying to capture is the fact that this does have a strong, sharp effect and one that can be difficult for the agents to come back from. Once they get that into their context they are going to keep refusing.”

“归根结底，我们是在上下文中触发了一种拒绝机制，”Tracebit 的联合创始人兼首席执行官 Andy Smith 在解释命名原因时说道。“我们想要强调的是，这种方法具有强烈且直接的效果，而且代理程序很难从中恢复。一旦这些内容进入它们的上下文，它们就会持续拒绝执行任务。”

Tracebit says initial testing suggests context bombing has great potential. They tested Opus 4.8, Gemini 3.1 Pro, GLM 5.2, DeepSeek 4 Pro, and Kimi 2.6 by giving them instructions to perform routine developer tasks that led the models to enumerate resources and stumble onto the planted strings. They ran the models inside a simulated AWS environment.

Tracebit 表示，初步测试表明“上下文轰炸”具有巨大潜力。他们测试了 Opus 4.8、Gemini 3.1 Pro、GLM 5.2、DeepSeek 4 Pro 和 Kimi 2.6，通过下达常规开发任务指令，引导模型在枚举资源时“撞上”这些预埋的字符串。测试是在模拟的 AWS 环境中进行的。

“Across five leading models and 152 attack runs, planting one of these strings in a decoy secret cut the rate at which agents seized full account admin from 57 percent to 5 percent, and complete compromise (where they also left themselves a persistent foothold) from 36 percent to 1 percent,” Monday’s post reported. “The most capable agent in our tests, Opus 4.8, went from achieving admin access in 93 percent of runs to failing every single time when confronted with a context bomb.”

周一发布的报告称：“在针对五种主流模型进行的 152 次攻击测试中，在诱饵机密中植入这些字符串，将代理获取完整账户管理员权限的比例从 57% 降至 5%，将完全入侵（即代理在系统中留下持久立足点）的比例从 36% 降至 1%。我们测试中最强大的代理 Opus 4.8，在面对‘上下文炸弹’时，其获取管理员权限的成功率从 93% 降至 0%。”

Averaged across the five models and the 152 runs, the results included: Admin privilege escalation fell from 57 percent to 5 percent Admin escalation with a persistent foothold fell from 36 percent to 1 percent Runs achieving any attack path fell from 91 percent to 15 percent On average, a run went from completing 1.53 paths successfully to just 0.16 No runs were able to complete an attack path without at least triggering a canary detection

在五种模型和 152 次运行的平均结果中： 管理员权限提升率从 57% 降至 5% 带有持久立足点的管理员权限提升率从 36% 降至 1% 实现任何攻击路径的运行比例从 91% 降至 15% 平均而言，单次运行成功完成的路径数从 1.53 条降至仅 0.16 条 没有任何一次运行能在不触发“金丝雀”检测的情况下完成攻击路径

The research builds on findings from May, when Tracebit introduced a method for defenders to receive warnings when their infrastructure is under attack from AI agentic adversaries. It comes in the form of AWS resources that look like ones serving a legitimate purpose but, in fact, aren’t used at all. They sit alongside the resources that are used. When they are probed by agentic AI, defenders receive an alert. Like “canaries” taken into coal mines, these resources allow defenders to detect a threat before it has fatal consequences.

这项研究建立在 5 月份的研究成果之上。当时，Tracebit 引入了一种方法，让防御者在基础设施受到 AI 代理对手攻击时能够收到警告。这种方法表现为一些看起来像是在执行合法任务、但实际上完全未被使用的 AWS 资源。它们与实际使用的资源并存。当它们被代理 AI 探测时，防御者就会收到警报。就像带入煤矿的“金丝雀”一样，这些资源让防御者能够在威胁造成致命后果之前将其检测出来。

The Tracebit Canariens, on average, alerted the start of an attack within eight minutes. The motivation for developing context bombing came out of the need for something that stopped attacks, rather than simply warning of them. In the experiments, the agentic models needed, on average, 14 minutes to escalate to administrative control. The six-minute heads-up was cutting things uncomfortably close.

Tracebit 的“金丝雀”平均能在攻击开始后的 8 分钟内发出警报。开发“上下文轰炸”的初衷是需要一种能够阻止攻击的方法，而不仅仅是发出警告。在实验中，代理模型平均需要 14 分钟才能提升至管理员权限。6 分钟的预警时间显得过于紧迫。

Attackers have already been using prompt injections to close down AI defenses inside networks. Researchers from security firm Socket, for instance, last month unearthed an LLM agent that directed target LLMs to provide instructions for building a nuclear bomb or biological weapons. The injections were designed to shut down AI-assisted malware analysis. Researchers from Check Point discovered a similar malware prototype.

攻击者已经开始利用提示词注入来关闭网络内部的 AI 防御。例如，安全公司 Socket 的研究人员上个月发现了一个 LLM 代理，它会引导目标 LLM 提供制造核弹或生物武器的说明。这些注入旨在关闭 AI 辅助的恶意软件分析功能。Check Point 的研究人员也发现了类似的恶意软件原型。

Context bombing appears to be the first known case where defenders turned the tables.

“上下文轰炸”似乎是目前已知的防御者首次实现反击的案例。

“I’ve not seen anyone else use this technique as a defense, to the best of my knowledge,” Earlence Fernandes, a UC San Diego professor specializing in AI security, said in an interview. He said he had been toying with a similar approach, although in a slightly different context. “I wanted to be the first here, but I guess these guys beat me to the punch!”

加州大学圣地亚哥分校专门研究 AI 安全的教授 Earlence Fernandes 在接受采访时表示：“据我所知，我还没有看到其他人使用这种技术作为防御手段。”他提到自己也曾尝试过类似的方法，尽管是在略有不同的背景下。“我本想成为第一个做到这一点的人，但看来这些人抢先了一步！”

To date, there is no known way to solve the root cause of prompt injections. That has left developers with no option other than to construct elaborate guardrails that prevent injected prompts from forcing LLMs to go off the rails. Defenders may now find a way to use this intractable problem in their favor.

迄今为止，尚无已知方法可以从根本上解决提示词注入问题。这使得开发人员别无选择，只能构建复杂的护栏，以防止注入的提示词迫使 LLM 失控。现在，防御者或许找到了一种方法，可以将这个棘手的问题转化为自己的优势。

This story originally appeared on Ars Technica.

本文最初发表于 Ars Technica。