The Pope’s AI Guy Is Worried About ‘Cartel’ Behavior Among Big Labs
The Pope’s AI Guy Is Worried About ‘Cartel’ Behavior Among Big Labs
教皇的 AI 顾问担忧大型实验室存在“卡特尔”行为
Recently, warnings from researchers inside the world’s largest artificial intelligence labs about the rapid pace of development have led to panic about the possibility that the technology could extinguish humanity. Those warnings are predicated on the idea that—propelled by ferocious competition and a rivalry between the US and China—labs are barreling toward superintelligent models whose abilities will far surpass those of humans.
最近，全球最大的人工智能实验室内部研究人员发出的警告称，技术发展的飞速步伐引发了人们的恐慌，担心该技术可能会导致人类灭绝。这些警告基于这样一种观点：在激烈的竞争以及中美博弈的推动下，各大实验室正全速冲向能力远超人类的超智能模型。
But according to Father Paolo Benanti, a priest who advises the Catholic Church on AI, the handwaving over superintelligence distracts from the need for open debate about how to constrain the companies building the technology. By casting the problem as so complex that only they can solve it, he says, the big labs exclude everyone else from the conversation.
但据为天主教会提供 AI 咨询的神父保罗·贝南蒂（Paolo Benanti）称，围绕超智能的夸夸其谈分散了人们的注意力，使人们忽略了就如何约束这些技术开发公司进行公开辩论的必要性。他说，大型实验室将问题描述得如此复杂，以至于只有他们能解决，这实际上将其他人排除在了对话之外。
“Ethics cannot be something that belongs to the board of trustees of a company. It’s something that has to be balanced, mitigated in debate in the public square.” Benanti tells WIRED. “We need democratic regulation.”
“伦理不能仅仅属于公司的董事会。它必须在公共广场的辩论中得到平衡和调解，”贝南蒂告诉《连线》（WIRED）杂志，“我们需要民主化的监管。”
In an essay published on the heels of the safety warnings, Anthropic CEO Dario Amodei called for new AI models to undergo mandatory third-party evaluations prior to release, and some sort of crackdown on distillation, a process where rival labs train their own AI on the outputs of proprietary models. Others have argued for a total ban on recursive self-improvement, where AI uses its advanced coding capabilities to help develop future iterations of itself.
在安全警告发布后不久，Anthropic 首席执行官达里奥·阿莫代（Dario Amodei）在一篇文章中呼吁，新的 AI 模型在发布前必须经过强制性的第三方评估，并对“蒸馏”（distillation）进行某种程度的打击——这是一种竞争对手实验室利用专有模型的输出结果来训练自身 AI 的过程。其他人则主张全面禁止递归自我改进，即 AI 利用其先进的编码能力来帮助开发自身的未来迭代版本。
Benanti is less interested in restricting methods for developing AI, which he fears could quash innovation, and more so in establishing standards that ensure models cannot slip from human control. Whatever shape those standards take, he says, they should not be designed by the labs.
贝南蒂对限制 AI 开发方法兴趣不大，他担心这会扼杀创新；他更关注建立标准，以确保模型不会脱离人类的控制。他说，无论这些标准采取何种形式，都不应由实验室自行设计。
“The narrative of the frontier labs is self-interested, to gain or maintain a position in the market,” says Benanti. When a small group of companies dictates the rules under which an industry operates, he says, “this has a really clear name: a cartel.”
“前沿实验室的叙事是出于自身利益，旨在获得或维持市场地位，”贝南蒂说。当一小群公司决定了一个行业的运作规则时，他说，“这有一个非常明确的名称：卡特尔（垄断联盟）。”
It’s difficult to know how seriously to take grand pronouncements about the capabilities of the latest and future AI models, says Benanti, owing to the overlapping economic, political, and personal incentives at play.
贝南蒂表示，由于涉及重叠的经济、政治和个人动机，很难判断对于最新和未来 AI 模型能力的宏大声明该有多认真。
While both OpenAI and Anthropic have supported calls for a slowdown in development, they’re also preparing to go public, and there are many billions of dollars at stake for whoever captures the market. The animosity between the US and China, where the most advanced AI models are being developed, also complicates the picture.
尽管 OpenAI 和 Anthropic 都支持放缓发展的呼吁，但它们也都在准备上市，对于任何占领市场的公司来说，这都涉及数十亿美元的利益。最先进 AI 模型开发地——中美两国之间的敌意，也使局势变得更加复杂。
Anthropic declined to comment. OpenAI and Google DeepMind did not respond to requests for comment.
Anthropic 拒绝置评。OpenAI 和 Google DeepMind 未回应置评请求。
Benanti also takes issue with the way certain corners of the AI industry talk about superintelligence, with the kind of reverence typically reserved for a god: as sophisticated beyond our understanding, practically omnipotent. (OpenAI CEO Sam Altman once said he aspires to build a “magic intelligence in the sky.”) That kind of framing, Benanti says, threatens to skew the discourse about the technology.
贝南蒂还对 AI 行业某些角落谈论超智能的方式表示异议，他们以通常留给上帝的敬畏态度来谈论它：认为它复杂到超出我们的理解，几乎无所不能。（OpenAI 首席执行官萨姆·奥特曼曾表示，他渴望构建一种“天上的魔法智能”。）贝南蒂说，这种框架可能会扭曲关于该技术的讨论。
“Are we talking about some-thing, or some-one? This is the real difference. It’s not a matter of faith, it’s a matter of mathematics,” he says. “We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation [going on].”
“我们谈论的是‘某物’还是‘某人’？这是真正的区别。这不是信仰问题，而是数学问题，”他说，“我们不是在创造下一个神。我们只是在自欺欺人，认为除了统计计算之外还有其他东西（在发生）。”
By raising questions about what separates man and machine, and how to shield future generations from the negative effects of technology, the advent of AI has the potential to create “a new role for religions in public debate,” says Benanti. “Living the challenge of the time is part of the mission of the Church.”
贝南蒂说，通过提出关于人与机器的区别，以及如何保护后代免受技术负面影响的问题，AI 的出现有可能为“宗教在公共辩论中创造一个新的角色”。“经历时代的挑战是教会使命的一部分。”
Though Benanti is a member of the priesthood, he does not formally represent the Holy See, the governing body of the Catholic Church. Instead, he counsels the church in his capacity as a scholar of philosophy, theology, and technology. Benanti tells WIRED he uses AI to help create tools and materials for his university students, but declined to specify how Pope Francis engages with the technology, if at all. (Benanti’s staff say he is bound by an “oath of secrecy.”)
虽然贝南蒂是神职人员，但他并不正式代表天主教会的行政机构——圣座。相反，他以哲学、神学和技术学者的身份为教会提供咨询。贝南蒂告诉《连线》，他使用 AI 来帮助为大学生创建工具和材料，但拒绝透露教皇方济各（Pope Francis）是如何接触该技术的（如果有的话）。（贝南蒂的工作人员表示，他受到“保密誓言”的约束。）
In May, Pope Francis published his first-ever encyclical, a letter that applies religious teachings to the problems of the day and clarifies the position of the Church. He chose artificial intelligence as its subject.
今年 5 月，教皇方济各发布了他有史以来第一份通谕，这是一封将宗教教义应用于当代问题并阐明教会立场的信函。他选择了人工智能作为主题。
In the letter, the Pope warned against the technocratic logic that runs through the AI industry, which presents “efficiency as the ultimate measure of value,” and casts human beings as “a project to be optimized.” He also called for international collaboration between rival superpowers and AI companies, who he said must set aside the “desire to secure geopolitical or commercial dominance” to ensure that the technology is developed in service of the common good. In one particularly prescient section, he warned of “the danger of humanity becoming a victim of its own achievements.”
在信中，教皇警告了贯穿 AI 行业的“技术官僚逻辑”，这种逻辑将“效率作为价值的终极衡量标准”，并将人类视为“一个待优化的项目”。他还呼吁竞争超级大国和 AI 公司之间进行国际合作，他说他们必须抛开“确保地缘政治或商业主导地位的欲望”，以确保技术的发展服务于共同利益。在一个特别有预见性的章节中，他警告了“人类成为自身成就受害者的危险”。
The flap over superintelligence and existential risk threatens to distract from the ambitions laid out in the encyclical, says Benanti, by encouraging people to forfeit their say in the direction the technology should take to a small group of mathematicians, computer scientists, and executives at the big labs.
贝南蒂说，围绕超智能和生存风险的争论可能会分散人们对通谕中所述抱负的注意力，因为它鼓励人们放弃对技术发展方向的发言权，转而将其交给一小群数学家、计算机科学家和大型实验室的高管。
“The first thing is, we need to have brakes, and a wheel,” says Benanti. “That means a significant position of control for human beings over the machine.”
“首先，我们需要有刹车和方向盘，”贝南蒂说，“这意味着人类必须对机器拥有重要的控制地位。”