Extinction-level capitalism: A citizen’s thoughts on AI risk

灭绝级资本主义：一位公民对人工智能风险的思考

AI is inherently political technology. If AI works as intended, it will gradually corrode our liberal democracy, risking an irreversible shift into another political and economic configuration. Among AI risks, this one deserves more consideration, because it requires no additional conditions like malign actors or AI malfunction. AI only needs to amplify existing trends, especially around concentration of capital. This damage will occur even assuming that in the near term, AI will broadly improve material well-being.

人工智能本质上是一种政治技术。如果人工智能按预期运行，它将逐渐侵蚀我们的自由民主制度，并可能导致不可逆转地转向另一种政治和经济格局。在人工智能的诸多风险中，这一点值得更多关注，因为它不需要恶意行为者或人工智能故障等额外条件。人工智能只需要放大现有的趋势，特别是围绕资本集中的趋势。即使假设人工智能在短期内能广泛改善物质福祉，这种损害依然会发生。

About MB

关于 MB

I’m a self-employed author, designer, programmer, and lawyer. In 2022, I learned that my own works were in the training datasets of generative-AI companies. In response, I invented the first set of lawsuits challenging the legality of these practices. I’m currently co-counsel for plaintiffs in a number of AI cases. Though I discuss certain legal issues below, I am not your lawyer, and nothing here is held out as legal advice. These are my personal views as a citizen and economic actor; I speak only for myself. This piece is typeset in Equity, Advocate, and Triplicate, fonts I designed. They can be licensed for your own polemics and pamphlets.

我是一名自由职业作家、设计师、程序员和律师。2022 年，我发现自己的作品被纳入了生成式人工智能公司的训练数据集。作为回应，我发起了首批挑战这些做法合法性的诉讼。目前，我在多起人工智能案件中担任原告的共同代理律师。虽然我在下文中讨论了某些法律问题，但我并非您的律师，本文内容也不构成任何法律建议。这些是我作为一名公民和经济参与者的个人观点；我仅代表我自己。本文使用了我设计的 Equity、Advocate 和 Triplicate 字体排版。您可以获得这些字体的授权，用于您自己的论战文章或宣传册。

Emergent effects

涌现效应

Two billion years ago, the rock layers comprising what is now called the Colorado Plateau began to form: first igneous and metamorphic rocks, followed by many layers of sedimentary rocks. About fifty million years ago, through tectonic action, this plateau gained thousands of feet of elevation. About five million years ago, a river began to flow. The river carried silt and debris, scraping out the beginnings of a canyon. The river deepened the canyon, exposing its walls to weather and erosional forces that widened the canyon further. Today the waterway is the Colorado River. The geological formation is the Grand Canyon.

二十亿年前，构成现今科罗拉多高原的岩层开始形成：首先是火成岩和变质岩，随后是多层沉积岩。大约五千万年前，通过构造运动，该高原抬升了数千英尺。大约五百万年前，一条河流开始流淌。河流携带泥沙和碎屑，冲刷出了峡谷的雏形。河流不断加深峡谷，使其岩壁暴露在风化和侵蚀作用下，从而进一步拓宽了峡谷。今天，这条水道就是科罗拉多河，而这一地质构造便是大峡谷。

The formation of the Grand Canyon required zero human agency. Zero technology. Zero coordination among the river, the land, and gravity. In that sense the Grand Canyon is an emergent effect: a complex, unforeseeable output arising from simpler inputs. But we would never wonder whether the river is sentient. Or whether the river cares about the dirt that it carries out of the canyon. The water is just doing what water does: flowing downhill. The dirt just happens to be in the way.

大峡谷的形成不需要任何人类干预，不需要任何技术，也不需要河流、土地和重力之间的任何协调。从这个意义上说，大峡谷是一种“涌现效应”：即由更简单的输入产生的一种复杂且不可预见的输出。但我们绝不会去怀疑河流是否有知觉，也不会去想河流是否在意它从峡谷中带走的泥土。水只是在做水该做的事：向低处流。而泥土恰好挡在了路上。

Inherently political technology

本质上的政治技术

Langdon Winner is a political theorist. Winner wrote the excellent and influential essay “Do Artifacts Have Politics?” (1980). Winner sought to debunk the traditional framing that “technologies are … neutral tools that can be used well or poorly, for good, evil, or something in between.” Instead, Winner proposes two ways that a technology can affect its political environment:

兰登·温纳（Langdon Winner）是一位政治理论家。他撰写了那篇出色且极具影响力的文章《人工制品有政治性吗？》（1980 年）。温纳试图驳斥那种认为“技术是……中性工具，可以被善用或恶用，用于行善、作恶或介于两者之间”的传统框架。相反，温纳提出了技术影响其政治环境的两种方式：

1. The technology is designed to have certain political effects. For example, the Great Firewall of China, a bundle of technological measures that limit Chinese citizens’ access to foreign information sources. Antipodally, the Tor Project intends to maximize user anonymity and thwart government intrusion.

2. 技术被设计为具有特定的政治效果。例如，中国的防火长城，这是一套限制中国公民访问外国信息源的技术措施。与之相反，Tor 项目旨在最大化用户匿名性并阻止政府入侵。

3. The technology is inherently political. This is Winner’s key analytic fulcrum. Winner describes two versions of inherently political technology. The first is where the technology “actually requires … a particular set of social conditions as [its] operating environment.” For instance, nuclear weapons: the only responsible way to possess such dangerous tech is to place it within “a centralized, rigidly hierarchical chain of command … the [atom] bomb must be authoritarian; there is no other way.” The second version is where the technology is “strongly compatible” with a certain political arrangement (even if not strictly required) and thus tends to tend to bring that arrangement to fruition.

4. 技术本质上是政治性的。这是温纳分析的核心支点。温纳描述了两种本质上具有政治性的技术。第一种是技术“实际上需要……一套特定的社会条件作为其运行环境”。例如核武器：拥有这种危险技术的唯一负责任方式是将其置于“一个集权的、严格等级森严的指挥链中……原子弹必须是威权主义的；别无他法。”第二种是技术与某种政治安排“高度兼容”（即使并非严格要求），因此往往会促成这种安排的实现。

As an example, Winner considers the mechanical tomato harvester. Developed at UC Davis in the 1950s, the machine was tremendously productive. But it was also expensive. Only well-capitalized tomato growers could afford it. The rest couldn’t compete. According to Winner, the number of California tomato growers dropped from ~4000 in the early 1960s to ~600 in 1973, costing ~32,000 jobs and the compounding negative effects on those communities. Winner summarizes:

温纳以机械番茄收割机为例。该机器于 20 世纪 50 年代在加州大学戴维斯分校开发，生产效率极高，但价格昂贵。只有资本雄厚的番茄种植者才买得起，其他人无法竞争。据温纳统计，加州的番茄种植者数量从 20 世纪 60 年代初的约 4000 家下降到 1973 年的约 600 家，导致约 32,000 个工作岗位流失，并对这些社区产生了连锁的负面影响。温纳总结道：

“What we see here … is an ongoing social process in which scientific knowledge, technological invention, and corporate profit reinforce each other in deeply entrenched patterns that bear the unmistakable stamp of political and economic power … opponents of innovations like the tomato harvester are made to seem ‘antitechnology’ or ‘antiprogress’. For the harvester is not merely the symbol of a social order that rewards some while punishing others; it is in a true sense an embodiment of that order.”

“我们在这里看到的是……一个持续的社会过程，其中科学知识、技术发明和企业利润在根深蒂固的模式中相互加强，这些模式带有政治和经济权力的明显印记……像番茄收割机这类创新的反对者被描绘成‘反技术’或‘反进步’。因为收割机不仅是一个奖励某些人而惩罚另一些人的社会秩序的象征；它在真正意义上是那种秩序的体现。”

Not merely the symbol—the embodiment. A facially neutral technological invention—here, a tomato harvester—can induce political effects. Those effects don’t arise from flaws in the technology. To the contrary—they arise from its efficacy.

不仅仅是象征，而是体现。一项表面上中性的技术发明——在这里是番茄收割机——可以引发政治效应。这些效应并非源于技术的缺陷。恰恰相反，它们源于技术的效能。

How are the political effects determined? Winner identifies two key early decisions. The first is the binary question of whether to pursue the technology at all. The second are choices about “the design or arrangement” of the technology. Winner cautions: “[t]o see the matter solely in terms of cost-cutting, efficiency, or the modernization of equipment is to miss a decisive element”. That is, the political effects can possibly be countered, but first they must be acknowledged.

政治效应是如何决定的？温纳指出了两个关键的早期决策。第一个是关于是否要追求该技术的二元问题。第二个是关于技术“设计或安排”的选择。温纳告诫说：“如果仅仅从成本削减、效率或设备现代化来看待这个问题，就会错过一个决定性的因素”。也就是说，政治效应或许可以被抵消，但首先必须承认它们的存在。

Of course, the best opportunity to choose wisely is before the technology is widely introduced, as capital and social investment tends to entrench it: “Because choices tend to become strongly fixed in material equipment, economic investment, and social habit, the original flexibility vanishes for all practical purposes once the initial commitments are made. In that sense tech…”

当然，做出明智选择的最佳时机是在技术被广泛引入之前，因为资本和社会投资往往会使其根深蒂固：“由于选择往往会牢固地固定在物质设备、经济投资和社会习惯中，一旦最初的承诺做出，最初的灵活性在实际意义上就会消失。从这个意义上说，技术……”