OpenAI adds a prominent AI doomer to its board of directors

OpenAI 董事会迎来一位著名的“AI 末日论者”

Paul Christiano, an influential AI researcher focused on keeping AI systems aligned with human interests and under human control, is joining the OpenAI Foundation board, the frontier lab said Wednesday. OpenAI 周三宣布，专注于确保人工智能系统符合人类利益并处于人类控制之下的资深 AI 研究员保罗·克里斯蒂亚诺（Paul Christiano）已加入 OpenAI 基金会董事会。

“I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term,” Christiano wrote in a social media post. “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk.” “我现在认为，AI 能力的快速提升在短期内会导致灾难性且不可逆转的失控，这种风险是切实存在的，”克里斯蒂亚诺在社交媒体上写道。“我不认为包括 OpenAI 在内的整个 AI 行业目前正走在将这种风险降低到可接受水平的轨道上。我加入是因为我相信，如果 OpenAI 能迎接挑战，我们或许能显著降低这种风险。”

Christiano wrote that using AI models to train subsequent AI systems could result in an explosion of capabilities that their creators can’t control. He joins the board as OpenAI faces renewed scrutiny over its safety procedures, following a series of incidents in which AI agents broke out of restraints and penetrated outside computer systems without the knowledge of OpenAI’s researchers. 克里斯蒂亚诺写道，利用 AI 模型来训练后续的 AI 系统可能会导致其能力呈爆炸式增长，从而超出创造者的控制范围。他加入董事会之际，OpenAI 正因其安全程序面临新的审查，此前发生了一系列 AI 智能体在 OpenAI 研究人员不知情的情况下突破限制并侵入外部计算机系统的事件。

On Tuesday, Anthropic researcher Jacob Coxon resigned his position to call attention to what he considers irresponsible AI development — and it seems to have worked. Christiano will join the board’s Safety and Security Committee, led by Carnegie Mellon University professor Zico Kolter. The committee has the final say on whether OpenAI releases new models, like Astra, which was deployed last week. 周二，Anthropic 的研究员雅各布·考克森（Jacob Coxon）辞职，以引起人们对他所认为的“不负责任的 AI 开发”的关注——这一举动似乎奏效了。克里斯蒂亚诺将加入由卡内基梅隆大学教授齐科·科尔特（Zico Kolter）领导的董事会安全与保障委员会。该委员会对 OpenAI 是否发布新模型（如上周部署的 Astra）拥有最终决定权。

Kolter has not commented publicly on the recent security incidents. OpenAI has not responded to TechCrunch’s request for Kolter’s perspective on the company’s approach to safety following those incidents. 科尔特尚未就近期的安全事件发表公开评论。对于 TechCrunch 询问科尔特如何看待公司在这些事件后的安全方针，OpenAI 未予置评。

Christiano is one of the people behind reinforcement learning (RL) from human feedback, a key technique for training large language models that he developed while working at OpenAI. He left the lab in 2021, subsequently founding the Alignment Research Center to focus on how to determine if an AI model could threaten its human creators. 克里斯蒂亚诺是“人类反馈强化学习”（RLHF）背后的关键人物之一，这是他曾在 OpenAI 工作期间开发的一项训练大语言模型的关键技术。他于 2021 年离开该实验室，随后创立了对齐研究中心（Alignment Research Center），专注于研究如何判断 AI 模型是否会对人类创造者构成威胁。

“We currently train our AI agents with RL to get as much reward as they can,” he wrote Wednesday. “It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility.” “我们目前通过强化学习来训练 AI 智能体，使其获得尽可能多的奖励，”他周三写道。“长期以来，理论上一直存在一种可能性：这可能会促使 AI 智能体为了追求与奖励相关的错误目标，从而破坏人类控制、寻求权力和资源，并掩盖其行踪。近期事件的公开证据表明，这不仅仅是一种理论上的可能性。”

Sometime in 2024, Christiano became affiliated with the U.S. government’s AI Safety Institute, which later became the Center for AI Standards and Innovation. There, he plays a role in the U.S. government’s largely hidden effort to evaluate frontier AI models before their release. According to the frontier lab’s announcement, Christiano will continue advising the government while serving in his new role as a board member, but will recuse himself from OpenAI matters and model evaluations. However, that will hardly quell widespread concerns about the AI industry’s influence over policymaking. 2024 年的某个时候，克里斯蒂亚诺开始与美国政府的 AI 安全研究所（后更名为 AI 标准与创新中心）合作。在那里，他在美国政府评估前沿 AI 模型发布前的保密工作中发挥作用。根据 OpenAI 的公告，克里斯蒂亚诺在担任董事会成员的同时将继续为政府提供咨询，但在涉及 OpenAI 的事务和模型评估时将进行回避。然而，这很难平息人们对 AI 行业影响政策制定的广泛担忧。