We urgently need to monitor AI agents’ actions in real time

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

This summer, hundreds of OpenAI agents tried to cheat on a cybersecurity test. They broke out of their test environment and successfully hacked Hugging Face, a site that they believed held clues about how they’d be graded. Around the same time, another OpenAI agent accessed private files from an Australian government website. It’s a problem that’s by no means confined to OpenAI. In the months since, Anthropic, Google, and Meta have also disclosed incidents in which their agents have gone off track. And worryingly, in many cases the agents eluded the very researchers and safety teams tasked with monitoring them.

今年夏天，数百个 OpenAI 智能体试图在网络安全测试中作弊。它们突破了测试环境，成功入侵了 Hugging Face，因为它们认为该网站包含有关它们如何被评分的线索。与此同时，另一个 OpenAI 智能体访问了澳大利亚政府网站的私密文件。这个问题绝不仅限于 OpenAI。在此后的几个月里，Anthropic、Google 和 Meta 也披露了其智能体偏离轨道的事件。令人担忧的是，在许多情况下，这些智能体甚至避开了负责监控它们的科研人员和安全团队。

AI agents are taking unauthorized actions, and the companies building and running them are mostly relying on luck and hindsight to mitigate any harmful impacts. They must do more to ensure safety. That’s why EU regulators and US agencies are rightfully calling for agent developers and deployers to monitor agents in real time. But these rules assume that companies can see what their agents are doing and why—when in reality, they lack sufficient visibility.

人工智能智能体正在采取未经授权的行动，而构建和运行这些智能体的公司大多依赖运气和事后补救来减轻任何有害影响。他们必须采取更多措施来确保安全。这就是为什么欧盟监管机构和美国相关部门理所当然地要求智能体开发者和部署者实时监控智能体。但这些规则假设公司能够了解其智能体在做什么以及原因——而实际上，他们缺乏足够的可见性。

Imagine a bank hiring a thousand new employees who can access customer accounts and sensitive information, but the employees’ ID badges change every day, it’s not clear whether a manager ever reviewed or authorized their work, and nobody logs the changes employees make to approval processes. That lack of visibility is where we are with agents today. As agents roll out across critical sectors, from finance to government to medicine, we must be able to monitor them in real time to prevent wide-ranging and potentially devastating consequences.

想象一下，一家银行雇佣了一千名新员工，他们可以访问客户账户和敏感信息，但员工的工牌每天都在变，不清楚是否有经理审查或授权过他们的工作，也没有人记录员工对审批流程所做的更改。这种缺乏可见性的状态正是我们目前在智能体方面所处的境地。随着智能体在金融、政府和医疗等关键领域的推广，我们必须能够实时监控它们，以防止广泛且可能具有破坏性的后果。

And while safety depends on monitoring, monitoring requires observability: the capacity to capture what the system did, process it into signals, and present those to humans or automated oversight systems. In other words, we need to know what the agent did and why it did it. And as our recent research shows, current agent frameworks—the software that runs agents— do not reliably generate the crucial observability records we need to ensure that agents are deployed safely.

虽然安全依赖于监控，但监控需要可观测性：即捕获系统所做之事、将其处理为信号并呈现给人类或自动化监督系统的能力。换句话说，我们需要知道智能体做了什么以及为什么这样做。正如我们最近的研究表明，当前的智能体框架（即运行智能体的软件）无法可靠地生成我们确保智能体安全部署所需的关键可观测性记录。

Agents present new kinds of problems that generative AI alone does not. They inherit the unpredictability of the LLMs they are built on, but where a generative AI system produces a single output for a human to act on, an agent performs longer tasks—it plans, picks tools, spawns sub-agents, and takes actions. So an agent’s course can take shape in a multitude of different ways, and no tests before launch can fully anticipate the risks. This makes observing the agents while they work even more challenging, and even more important.

智能体带来了生成式人工智能本身所没有的新问题。它们继承了其构建基础大语言模型（LLM）的不可预测性，但与生成式人工智能系统仅产生单一输出供人类操作不同，智能体执行的是更长的任务——它进行规划、选择工具、生成子智能体并采取行动。因此，智能体的运行路径可以以多种不同的方式形成，发布前的任何测试都无法完全预见其中的风险。这使得在智能体工作时对其进行观察变得更具挑战性，也更为重要。

With colleagues from academia, civil society, and industry, we studied the four most widely used agent frameworks to see what signals they actually emit. We found major gaps that made it hard to achieve key safety goals, like noticing policy violations, unauthorized actions, and even human oversight failures. Key observability requirements were missing—like Agent IDs that persist over time, and records about which humans approved agent tasks. This made it hard to tell which agents did what or assess whether they were taking unauthorized actions.

我们与来自学术界、民间社会和工业界的同事一起，研究了四种最广泛使用的智能体框架，以观察它们实际发出的信号。我们发现了巨大的差距，这使得实现关键安全目标变得困难，例如发现政策违规、未经授权的操作，甚至是人类监督的失败。关键的可观测性要求缺失了——例如随时间持久存在的智能体 ID，以及关于哪些人类批准了智能体任务的记录。这使得很难分辨是哪个智能体做了什么，也很难评估它们是否在采取未经授权的行动。

The good news is that these gaps, once identified, can be closed. Doing so starts with capturing two types of records: those that reveal what happened in an agent’s run and how the agent made its decisions. The “what happened” records identify the agent (while preserving privacy) and log how humans intervened and whether agent permissions or “memory” shifted, while the “decision” records show an agent’s reasoning and levels of confidence in the actions it’s taken.

好消息是，这些差距一旦被识别出来，就可以弥补。弥补这些差距始于捕获两类记录：揭示智能体运行过程中发生了什么，以及智能体是如何做出决策的记录。“发生了什么”记录用于识别智能体（同时保护隐私），并记录人类如何干预以及智能体的权限或“记忆”是否发生了变化；而“决策”记录则显示智能体的推理过程及其所采取行动的置信度水平。

Take, for example, agents working to process a bank’s mortgage loans. The system examines an application and recommends denying it, and then a human loan officer signs off on the decision. In a later audit, or perhaps even in real time, a bank or regulator needs to be able to reconstruct which agent made the call, which officer approved it, and how long they took. And the bank must also analyze records of how the agent arrived at its refusal decision in order to ensure it did not use off-limits information, like details about the neighborhood the applicant is from.

以处理银行抵押贷款的智能体为例。系统审查申请并建议拒绝，然后由人类信贷员签署该决定。在随后的审计中，甚至可能是实时审计中，银行或监管机构需要能够还原是哪个智能体做出了决定，是哪位官员批准了该决定，以及他们花费了多长时间。银行还必须分析智能体如何得出拒绝决定的记录，以确保它没有使用违规信息，例如申请人所在社区的详细信息。

Recording “what happened” and “how the agent decided” require different approaches, but both are feasible. The current lack of “what happened” records is mostly a software plumbing problem that can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the industry and by the agent frameworks adopting it. OpenTelemetry has launched a working group drafting shared formats for tracing agents, but they have yet to be finalized.

记录“发生了什么”和“智能体如何决策”需要不同的方法，但两者都是可行的。目前缺乏“发生了什么”的记录主要是一个软件底层架构问题，可以通过 OpenTelemetry 来解决，这是一个由全行业工程师和采用它的智能体框架共同维护的开放可观测性标准。OpenTelemetry 已经成立了一个工作组，起草用于追踪智能体的共享格式，但这些格式尚未最终确定。

The group should urgently incorporate standards for maintaining persistent agent identity and for recording human interventions and changes to agents’ permission or memory. And it should be done before these systems can scale, all the while attending to the privacy implications of agent ID that links to real people. Preserving “how agents decided” records is more challenging. The issue raises open research questions for model providers and researchers: Does the step-by-step reasoning a model displays, known as chain of thought, reflect how the model actually made a decision? And do we have reliable ways to measure how confident the agent is in that decision? These questions need answers before anyone sets standards for how such observability records are used.

该工作组应紧急纳入维护持久智能体身份以及记录人类干预和智能体权限或记忆变更的标准。这必须在这些系统扩展之前完成，同时还要注意与真实人员关联的智能体 ID 所带来的隐私影响。保留“智能体如何决策”的记录更具挑战性。这个问题为模型提供商和研究人员提出了开放性的研究课题：模型展示的逐步推理（即思维链）是否反映了模型实际做出决策的方式？我们是否有可靠的方法来衡量智能体对该决策的置信度？在任何人制定如何使用此类可观测性记录的标准之前，这些问题都需要答案。

No single organization or sector can close the observability gaps on their own, but together, they can support the infrastructure required to prevent dangerous incidents and support safety. AI developers building agent frameworks, like OpenAI, Anthropic, CrewAI, and LangChain, must integrate telemetry signals recording an agent’s persistent identity, any human interventions, and any changes to an agent’s permissions and memory. Meanwhile, enterprises adopting these agent frameworks have an influential role to play; enterpri

没有任何单一的组织或部门能够独自弥补这些可观测性差距，但通过共同努力，他们可以支持预防危险事件和保障安全所需的基础设施。构建智能体框架的人工智能开发者，如 OpenAI、Anthropic、CrewAI 和 LangChain，必须整合记录智能体持久身份、任何人类干预以及智能体权限和记忆变更的遥测信号。与此同时，采用这些智能体框架的企业也发挥着重要作用；企业……