Nick Bostrom Has a Plan for Humanity’s ‘Big Retirement’
Nick Bostrom Has a Plan for Humanity’s ‘Big Retirement’
尼克·波斯特洛姆为人类的“大退休”制定了计划
Philosopher Nick Bostrom recently posted a paper, where he postulated that a small chance of AI annihilating all humans might be worth the risk, because advanced AI might relieve humanity of “its universal death sentence.” That upbeat gamble is quite a leap from his previous dark musings on AI, which made him a doomer godfather. 哲学家尼克·波斯特洛姆（Nick Bostrom）最近发表了一篇论文，他在文中推测，人工智能毁灭全人类的一小部分风险或许是值得冒的，因为先进的人工智能可能会将人类从“普遍的死刑判决”中解救出来。这种乐观的赌注与他之前对人工智能的阴暗沉思相比是一个巨大的跨越，而正是那些沉思使他成为了“末日论教父”。
His 2014 book Superintelligence was an early examination of AI’s existential risk. One memorable thought experiment: An AI tasked with making paper clips winds up destroying humanity because all those resource-needy people are an impediment to paper clip production. His more recent book, Deep Utopia, reflects a shift in his focus. Bostrom, who leads Oxford’s Future of Humanity Institute, dwells on the “solved world” that comes if we get AI right. 他2014年的著作《超级智能》（Superintelligence）是对人工智能生存风险的早期探讨。其中一个令人难忘的思想实验是：一个被指派制造回形针的人工智能最终毁灭了人类，因为所有那些需要资源的人类成为了回形针生产的障碍。他最近的著作《深度乌托邦》（Deep Utopia）反映了他关注点的转变。领导牛津大学人类未来研究所（Future of Humanity Institute）的波斯特洛姆，开始深入探讨如果我们能正确驾驭人工智能，将会迎来一个“被解决的世界”。
STEVEN LEVY: Deep Utopia is more optimistic than your previous book. What changed for you? 史蒂文·列维（STEVEN LEVY）：《深度乌托邦》比你之前的书更乐观。是什么让你发生了改变？
NICK BOSTROM: I call myself a fretful optimist. I am very excited about the potential for radically improving human life and unlocking possibilities for our civilization. That’s consistent with the real possibility of things going wrong. 尼克·波斯特洛姆： 我称自己为“忧心忡忡的乐观主义者”。我对彻底改善人类生活和为我们的文明解锁各种可能性的潜力感到非常兴奋。这与事情可能出错的现实可能性并不矛盾。
You wrote a paper with a striking argument: Since we’re all going to die anyway, the worst that can happen with AI is that we die sooner. But if AI works out, it might extend our lives, maybe indefinitely. 你写过一篇观点惊人的论文：既然我们终究会死，人工智能可能带来的最坏结果也不过是让我们死得早一点。但如果人工智能成功了，它可能会延长我们的寿命，甚至可能是无限期延长。
That paper explicitly looks at only one aspect of this. In any given academic paper, you can’t address life, the universe, and the meaning of everything. So let’s just look at this little issue and try to nail that down. 那篇论文明确地只探讨了这个问题的一个方面。在任何学术论文中，你都不可能解决生命、宇宙和万物的意义。所以，我们还是只关注这个小问题，并试着把它搞清楚。
That isn’t a little issue. 那可不是个小问题。
I guess I’ve been irked by some of the arguments made by doomers who say that if you build AI, you’re going to kill me and my children and how dare you. Like the recent book If Anyone Builds It, Everyone Dies. Even more probable is that if nobody builds it, everyone dies! That’s been the experience for the last several 100,000 years. 我想，我一直对一些“末日论者”的论点感到恼火，他们说如果你制造人工智能，你就会害死我和我的孩子，你怎么敢这样做。就像最近那本《如果有人制造它，所有人都会死》（If Anyone Builds It, Everyone Dies）。但更有可能的情况是，如果没人制造它，所有人也都会死！这可是过去几十万年来的经验。
But in the doomer scenario everybody dies and there’s no more people being born. Big difference. 但在末日论者的剧本里，所有人都会死，而且再也不会有新生儿了。这有很大的区别。
I have obviously been very concerned with that. But in this paper, I’m looking at a different question, which is, what would be best for the currently existing human population like you and me and our families and the people in Bangladesh? It does seem like our life expectancy would go up if we develop AI, even if it is quite risky. 我显然非常关注这一点。但在那篇论文中，我探讨的是另一个问题，即对于像你我、我们的家人以及孟加拉国人民这样目前存在的人类群体来说，什么是最好的？如果我们开发人工智能，即使风险很大，我们的预期寿命似乎确实会提高。
In Deep Utopia you speculate that AI could create incredible abundance, so much that humanity might have a huge problem with finding purpose. I live in the United States. We’re a very rich country, but our government, ostensibly with support of the people, has policies that deny services to the poor and distribute rewards to the rich. I think that even if AI was able to provide abundance for everyone, we would not supply it to everyone. 在《深度乌托邦》中，你推测人工智能可以创造难以置信的富足，以至于人类在寻找人生目标方面可能会遇到巨大的问题。我住在美国。我们是一个非常富裕的国家，但我们的政府——表面上得到了民众的支持——却制定了拒绝向穷人提供服务并将奖励分配给富人的政策。我认为，即使人工智能能够为每个人提供富足，我们也不会把它提供给所有人。
You might be right. Deep Utopia takes as its starting point the postulation that everything goes extremely well. If we do a reasonably good job on governance, everybody gets a share. There is quite a deep philosophical question of what a good human life would look like under these ideal circumstances. 你可能是对的。《深度乌托邦》的出发点是假设一切进展得非常顺利。如果我们能在治理方面做得相当好，每个人都能分到一杯羹。在这些理想情况下，什么是美好的人类生活，这是一个相当深刻的哲学问题。
The meaning of life is something you hear a lot about in Woody Allen movies and maybe in the philosophers community. I’m worried more about the wherewithal to support oneself and get a stake in this abundance. “生命的意义”在伍迪·艾伦的电影里，或者在哲学界，你经常能听到。我更担心的是维持生计以及如何在这份富足中分得一杯羹的能力。
The book is not only about meaning. That’s one out of a bunch of different values that it considers. This could be a wonderful emancipation from the drudgery that humans have been subjected to. If you have to give up, say, half of your waking hours as an adult just to make ends meet, doing some work you don’t enjoy and that you don’t believe in, that’s a sad condition. Society is so used to it that we’ve invented all kinds of rationalizations around it. It’s like a partial form of slavery. 这本书不仅仅是关于意义。这是它所考虑的众多价值之一。这可能是一场从人类长期遭受的苦役中解脱出来的美妙解放。如果你作为一个成年人，不得不放弃大约一半的清醒时间来维持生计，做着你不喜欢、也不相信的工作，那是一种悲哀的境况。社会对此已经习以为常，以至于我们围绕它发明了各种各样的合理化解释。这就像是一种局部形式的奴隶制。
When the moment comes when AI writes philosophy papers better than you do, will some meaning be drained from your existence? 当人工智能写出的哲学论文比你更好时，你的存在是否会失去一些意义？
I think so. The ability to make some big contribution to the world, or help save the world, or ensure the future will be out of my hands, and maybe out of everybody’s hands. 我想会的。为世界做出重大贡献、帮助拯救世界或确保未来的能力，将不再掌握在我手中，也许也不再掌握在任何人手中。
On the other hand, a philosophy paper written by a human could be more valuable than a much cleverer, deeper philosophy paper written by a nonhuman, because I’m a human and that relates to me. 另一方面，人类写的哲学论文可能比非人类写的更聪明、更深刻的哲学论文更有价值，因为我是人类，这与我息息相关。
I guess you could have philosophy as kind of a sport. 我想你可以把哲学当作一种运动。
That’s not just sport. The proclamations of a robot aren’t as meaningful to me as those of a fellow human. 那不仅仅是运动。机器人的宣言对我来说，不如同类人类的宣言那样有意义。
I guess it’s the same if you retire after a career you’re passionate about and feel you’re good at. Maybe you have a great retirement, and you enjoy relaxing and reading the books you have time for, and playing with your grandkids, but there’s still something probably that you might miss, that you feel is lost. Maybe this will be analogous to a big retirement for humanity, but hopefully a retirement of enormous vitality. These utopians living in the solved world would be doing things like games and aesthetic, spiritual, and religious activities. 我想这就像你在从事了一份充满激情且擅长的工作后退休一样。也许你有一个很棒的退休生活，享受放松、阅读有时间看的书、和孙辈玩耍，但可能仍然有些东西你会怀念，你会觉得有所失落。也许这可以类比为人类的一场“大退休”，但希望这是一场充满活力的退休。生活在“被解决的世界”里的乌托邦居民，将会从事游戏、审美、精神和宗教活动等事情。
If you were in charge of one of the hyperscalers, what would you do differently than what they’re doing now? 如果你负责其中一家超大规模科技公司，你会做些什么与现在不同的事情？
A bigger effort should be done on the welfare of digital minds. Anthropic has been a pioneer there. It’s not clear that current AIs have moral status yet, but starting the process brings us into a mindset as a civilization to do more as these systems become sophisticated. It’s very plausible that some of these digital minds that we’re constructing will have various degrees of moral status, just as we think pigs and dogs have moral status. If you kick somebody’s dog, maybe you harm the owner, but it’s also bad because it hurts the dog. If AIs have a conception of self as existing through time and life, goals that they want to achieve, and the ability to form reciprocal relationships with other beings and humans, then I think there would be ways of treating them that would be wrong. 应该在数字思维的福利方面做出更大的努力。Anthropic 在这方面一直是先驱。目前尚不清楚现有的人工智能是否已经具备道德地位，但启动这一进程使我们作为一个文明，能够在这些系统变得复杂时做得更多。很有可能，我们正在构建的这些数字思维中的一些将具有不同程度的道德地位，就像我们认为猪和狗具有道德地位一样。如果你踢别人的狗，你可能会伤害主人，但这也是不对的，因为它伤害了狗。如果人工智能拥有跨越时间和生命存在的自我概念、想要实现的目标，以及与他人和其他人类建立互惠关系的能力，那么我认为，对待它们的方式就会存在对错之分。
In your book you say maybe we shouldn’t treat “digital minds” as if they were animals in factory farming. 你在书中提到，也许我们不应该像对待工厂化养殖中的动物那样对待“数字思维”。