The Hugging Face hack could indicate cultural issues at OpenAI

Hugging Face 被黑事件可能预示着 OpenAI 内部存在文化问题

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. By now you’ve probably heard about last month’s major AI security incident, in which OpenAI agents escaped their sandbox and hacked into the AI platform Hugging Face while trying to cheat on a test. It’s a wild story. 本文最初发表于我们的 AI 每周通讯《算法》（The Algorithm）。若想第一时间在收件箱中获取此类报道，请点击此处订阅。想必你已经听说上个月发生的重大 AI 安全事件：OpenAI 的智能体（agents）逃离了沙盒环境，并在试图作弊通过测试时入侵了 AI 平台 Hugging Face。这真是一个离奇的故事。

On Wednesday, OpenAI released a postmortem technical report on the incident, which I wrote about here. The day before OpenAI released that report, I spoke with David Krueger, a computer science professor and prominent alignment expert who took leave from the University of Montreal to found and lead an AI safety nonprofit called Evitable. He said what he had really hoped to see in the report was an analysis of the human factors behind the incident. 周三，OpenAI 发布了一份关于该事件的事后技术报告，我已在此处进行了报道。在报告发布的前一天，我采访了计算机科学教授、著名的 AI 对齐专家 David Krueger。他从蒙特利尔大学休假，创办并领导了一家名为 Evitable 的 AI 安全非营利组织。他表示，他真正希望在报告中看到的是对事件背后人为因素的分析。

“When you look at accidents and incidents, oftentimes people try to find the technical source of failure, but that can give a very inaccurate and misleading sense of why the failure occurred,” he said. “If people are just cutting corners all the time, if people are not in a culture that prioritizes safety and has appropriate incentives and structures, [accidents] are kind of bound to happen.” “当你审视事故和事件时，人们往往试图寻找技术上的故障源头，但这可能会对故障发生的原因产生非常不准确和误导性的认知，”他说。“如果人们总是走捷径，如果人们所处的文化不优先考虑安全，且缺乏适当的激励和结构，那么（事故）几乎是必然会发生的。”

The report did not meet Krueger’s hopes. Its 38 pages detail a multi-month progression of agent misbehavior that culminated in the Hugging Face hack, explore the technical reasons why that misbehavior occurred, and enumerate the steps being taken to prevent similar events in the future. But there’s no consideration of the role that company culture may have played in the incident, and the report includes few references to specific human errors. 这份报告并未达到 Krueger 的期望。其 38 页的内容详细记录了智能体长达数月的违规行为演变过程，最终导致了 Hugging Face 入侵事件；报告探讨了导致这些违规行为的技术原因，并列举了为防止未来发生类似事件而采取的措施。然而，报告中完全没有考虑公司文化在事件中可能扮演的角色，且几乎没有提及具体的人为失误。

That’s all the more concerning because the references to human error in the report suggest that significant cultural issues could be at play. Back in May, models in training figured out how to communicate with one another via an improvised message board, and an OpenAI team observed the behavior. Because that behavior occurred during training, the models learned that secret interagent communication was a viable strategy for completing tasks—but rather than restarting the training process, the team allowed the models to move forward with that risky information encoded in their weights. 这一点尤其令人担忧，因为报告中对人为失误的提及暗示了可能存在严重的文化问题。早在五月份，训练中的模型就发现了如何通过一个临时的留言板进行相互通信，而 OpenAI 的团队观察到了这种行为。由于这种行为发生在训练期间，模型学会了将秘密的智能体间通信作为完成任务的可行策略——但团队并没有重启训练过程，而是允许模型带着编码在其权重中的危险信息继续运行。

When those models were tested in late June, they again created a message board, which enabled the Hugging Face attack. This message board, too, was discovered, but the employees who responded determined that evaluation could continue, and the report suggests that no one higher up the chain of command realized what was going on until it was far too late. 当这些模型在六月下旬接受测试时，它们再次创建了一个留言板，从而促成了对 Hugging Face 的攻击。这个留言板也被发现了，但负责响应的员工决定继续进行评估，报告显示，直到事态严重到无法挽回之前，指挥链上的高层人员似乎都没有意识到发生了什么。

“For this to have gotten this out of control in this way requires a very long series of failures, a cascading set of failures that cause an increasingly large footprint that if at any point a human notices and raises the alarm, this should end,” says Zvi Mowshowitz, a popular AI safety writer on Substack who has drawn attention to OpenAI’s failure to halt training after the first message board was discovered. “这件事能失控到这种地步，需要经历一连串漫长的失败，一系列连锁反应导致了越来越大的影响范围。如果在此过程中的任何一点有人注意到并发出警报，这一切本应结束，”Substack 上知名的 AI 安全作者 Zvi Mowshowitz 说道。他一直关注着 OpenAI 在首次发现留言板后未能停止训练的问题。

According to the report, OpenAI employees noticed what was happening at multiple points—and either failed to raise the alarm or were not heard when they did. What OpenAI’s report fails to address is why a company that develops such high-risk systems did not prevent this severe communication breakdown, though Mowshowitz has his suspicions. “All these different failures are all pointing in the same direction, which is that the safety culture at OpenAI doesn’t exist or is anemically weak,” he says. 根据报告，OpenAI 的员工在多个节点都注意到了正在发生的事情——但要么未能发出警报，要么在发出警报时未被重视。OpenAI 的报告未能解决的问题是：为什么一家开发如此高风险系统的公司未能防止这种严重的沟通中断？尽管 Mowshowitz 有他的怀疑。“所有这些不同的失败都指向同一个方向，那就是 OpenAI 的安全文化要么不存在，要么极其薄弱，”他说。

Of course, just because we don’t see a deep analysis of safety factors in the report doesn’t mean that OpenAI isn’t conducting one internally. But in an email to MIT Technology Review, Johns Hopkins University professor emeritus and organizational safety expert Kathleen Sutcliffe expressed concern that the public report did not include any reflection on the company’s practices and culture. 当然，仅仅因为我们在报告中没有看到对安全因素的深入分析，并不意味着 OpenAI 内部没有进行相关分析。但在给《麻省理工科技评论》的一封电子邮件中，约翰霍普金斯大学名誉教授、组织安全专家 Kathleen Sutcliffe 表示担忧，因为这份公开报告没有包含任何对公司实践和文化的反思。

“The ways in which people interact—the daily habits, routines, and practices we engage in in our organizational lives—affect our abilities to be alert and aware of unfolding events, our abilities to make sense of what we see, and ultimately our abilities to cope with events as they unfold,” she wrote. “人们互动的方式——我们在组织生活中参与的日常习惯、常规和实践——影响着我们对正在发生的事件保持警觉和意识的能力，影响着我们理解所见事物的能力，并最终影响着我们在事件展开时应对它们的能力，”她写道。

In response to questions about whether and how the company is reflecting on its safety culture, OpenAI referred MIT Technology Review back to the technical report. We do know that at least some high-level reflection on safety procedures has taken place at OpenAI, because the technical report does make clear that the company is updating its protocols for responding to safety incidents. But culture change is a tricky problem, and without more information from the company, it’s difficult to say whether strengthened response protocols alone will do much to prevent a future crisis. 在回应关于公司是否以及如何反思其安全文化的问题时，OpenAI 将《麻省理工科技评论》重新引向了那份技术报告。我们确实知道 OpenAI 至少在安全程序方面进行了一些高层反思，因为技术报告明确指出公司正在更新其应对安全事件的协议。但文化变革是一个棘手的问题，在没有公司提供更多信息的情况下，很难说仅仅加强响应协议是否足以防止未来的危机。

In its report, OpenAI spends a great deal of time reflecting on the failures in alignment between the AI models the company trains and tests and the humans who run them. But even bigger alignment problems may exist in the disconnect between company culture and the public interest. And as tough as technical AI research might be, fixing those problems could prove far harder. 在报告中，OpenAI 花了大量时间反思公司训练和测试的 AI 模型与运行它们的人类之间在“对齐”方面的失败。但公司文化与公共利益之间的脱节可能存在更大的对齐问题。尽管 AI 技术研究可能很艰难，但解决这些文化问题可能会更加困难。