Dario, Please

Dario Amodei, the CEO of Anthropic, recently published a blog post titled We Must Pace the Frontier and it is a load of bullshit, with a grim goal of regulating open weight models and giving the frontier labs an antitrust waiver.

Anthropic 的首席执行官 Dario Amodei 最近发表了一篇题为《我们必须放慢前沿步伐》（We Must Pace the Frontier）的博文，这简直是一派胡言。其阴暗的目的在于监管开源权重模型，并为前沿实验室争取反垄断豁免权。

Dario starts off with a claim of “AI will cure most major diseases in the next 5-10 years” and makes it personal. He talks about his father dying of a disease that was cured only years later and his own battle with cancer which he remarks was incurable 50 years ago. He also says AI will accelerate economic growth rates, create a world of abundance and empowerment, usher in renaissance of democracy and freedom. This largely reads as some kind of out-of-touch Silicon Valley, spends-a-lot-of-time-on-LessWrong, rich person’s idea of a future.

Dario 开篇便声称“人工智能将在未来 5-10 年内治愈大多数重大疾病”，并将其个人化。他谈到了父亲死于一种几年后就被治愈的疾病，以及他自己与癌症的斗争——他提到这种病在 50 年前是无法治愈的。他还表示，人工智能将加速经济增长，创造一个富足和赋权的世界，并迎来民主与自由的复兴。这读起来很大程度上像是一个脱离现实的硅谷富豪，一个整天泡在 LessWrong 论坛上的人，对未来的某种臆想。

Let me take this from the top. US labs are continuing to throw caution to the wind and be reckless. OpenAI does not seem to have a handle on things and they were caught three times recently hacking into public facing internet infrastructure. In Dario’s own essay, he alludes to “incidents” at Anthropic as well. With this pretext, Dario asks a lot from us. He wants open weight models to be regulated, distillation be dealt with a heavy hand, hand him an antitrust waiver, handicap China in multiple ways, essentially regulate themselves and a gentlemen’s agreement to slow down. All of this of course, comes in a package of extreme fear mongering to the detriment of our collective future and potential catalyst AI as a whole could be. They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us. This time around, it is imminent AGI, RSI and all of it turning rogue. Dario self-anoints his company and their close rival OpenAI as the stewards.

让我从头说起。美国的实验室继续将谨慎抛诸脑后，行事鲁莽。OpenAI 似乎无法掌控局面，最近三次被抓到入侵面向公众的互联网基础设施。在 Dario 自己的文章中，他也暗示了 Anthropic 内部也存在“事故”。以此为借口，Dario 向我们提出了诸多要求。他希望监管开源权重模型，对模型蒸馏采取严厉手段，要求给予他反垄断豁免权，在多方面限制中国，本质上是让他们进行自我监管，并达成一个放慢速度的“绅士协定”。当然，所有这些都包裹在极端的恐慌营销之中，损害了我们的集体未来以及人工智能作为整体可能带来的潜在催化作用。他们一次又一次地证明自己不可信，然而，他们主要的要求却是“信任我们，只信任我们”。这一次，他们抛出的理由是迫在眉睫的 AGI（通用人工智能）、RSI（递归自我改进）以及这一切可能带来的失控。Dario 自封他的公司及其竞争对手 OpenAI 为这些技术的“管家”。

Diseases, Prosperity, Abundance, then Freedom and Democracy??? Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.

疾病、繁荣、富足，然后是自由和民主？？？Anthropic 对生物学及相关研究的使用进行了限制。在他们最新的威胁情报报告中，他们谈到了如何检测并封禁了利用 Claude 系列模型进行可怕活动的恶意行为者。不得不承认，这是一个危险的边缘，而他们在检测和封禁滥用行为方面似乎做得不错。但仔细审视一下正在发生的事情：这种“万灵药”对你我来说是被封锁的，但 Anthropic 却雇佣生物学家，建立湿实验室，想要将发现据为己有。我在上一篇文章中已经提到过这一点。

In my view, there are billions of people with actual intelligence we have not managed to train or nurture. They will always remain victims of their circumstances. Tuberculosis has been curable for decades now, yet a million people die of it every year. Of course AGI will solve the distribution in a jiffy. To skirt around this uncomfortable truth, the goal is ASI/AGI/RSI and what not. A silver bullet for every problem, a noble pursuit, it may appear on the surface.

在我看来，有数十亿拥有真正智慧的人，我们却未能对其进行培养或教育。他们将永远是环境的受害者。肺结核几十年来一直是可以治愈的，但每年仍有百万人死于此病。当然，AGI 会在瞬间解决分配问题（讽刺）。为了回避这个令人不安的事实，他们的目标变成了 ASI（超级人工智能）/AGI/RSI 等等。表面上看，这似乎是解决所有问题的灵丹妙药，是一项崇高的追求。

Don’t even get me started on the prosperity and abundance bullshit. Abundance for the shareholders perhaps.

别跟我提什么繁荣和富足的鬼话。那可能只是股东们的富足吧。

I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie.

我不知道自由和民主与人工智能及前沿实验室有什么关系。除非 Dario 是《新世纪福音战士》的粉丝，梦想着由“三贤人”（Magi）超级计算机来管理政府。说实话，花 200 美元就能买到自由和民主，听起来确实很诱人。

Serious Economic Disruption / Race to the Bottom / Race to the Top

严重的经济破坏 / 逐底竞争 / 逐顶竞争

I feel like Dario is torn. He wants Anthropic to have this bad boy street cred of wielders of this crazy power, yet at the same time, he wants to make it seem like they are the cautious ones, always being faced with a trolley problem at every turn. Deaths from economic disruption and loss of jobs is okay, but deaths from a potential bioweapon is not. Remember this man has been saying software development will be solved in “6-12 months” forever now.

我觉得 Dario 很纠结。他既希望 Anthropic 拥有那种掌握疯狂力量的“坏小子”街头信誉，同时又想表现得他们是谨慎的一方，时刻面临着电车难题。因经济破坏和失业导致的死亡是可以接受的，但因潜在生物武器导致的死亡却不行。别忘了，这个人一直以来都在说软件开发将在“6-12 个月内”被解决。

He then gives it to us straight. “Race to the bottom” makes all the risks he pointed out more acute. Notice he does not say, the race to the bottom will be the end of his company. He instead wants a race to the “top”. Where labs will compete for safety.

然后他直言不讳。“逐底竞争”使他指出的所有风险变得更加尖锐。注意，他并没有说逐底竞争会导致他的公司倒闭。相反，他想要的是一场“逐顶竞争”，即实验室之间在安全性上进行竞争。

Incidentally, the most documented “race to the bottom” instance happened just days before. Upon hearing rumours about Anthropic close to or solving one or two Millennium problems, OpenAI threw tens of millions in compute, a training checkpoint, thousands of agents at it. It is also alleged that OAI stole the work of two mathematicians on a related problem, in the same narrow corner almost nobody else was working on. They published a proof of a forced variant of Navier-Stokes. I’m not a mathematician, but I have seen enough of Sam Altman’s antics to not take anything that comes out of him or his company at face value.

顺便提一下，最典型的“逐底竞争”案例就在几天前发生了。在听到关于 Anthropic 即将解决或已经解决一两个千禧年大奖难题的传闻后，OpenAI 投入了数千万美元的算力、一个训练检查点以及数千个智能体。还有人指控 OpenAI 窃取了两名数学家在相关问题上的研究成果，而该领域几乎无人涉足。他们发布了一个关于纳维-斯托克斯方程强制变体的证明。我不是数学家，但我见过足够多 Sam Altman 的把戏，不会轻信他和他的公司发布的任何东西。

Two Things that have Dario Scared: RSI and OAI-HF Incident

让 Dario 感到恐惧的两件事：RSI 和 OAI-HF 事件

OpenAI and the seller of shovels, Jensen Huang have claimed AGI has arrived with the release of GPT-6 Astra. RSI is the talk of the town now. LLMs or agents developing the next generation of LLMs with little to no human input. Amodei says it is happening across labs. I’ll believe it when there is actually some proof.

OpenAI 和“卖铲人”黄仁勋声称，随着 GPT-6 Astra 的发布，AGI 已经到来。RSI 现在成了热门话题，即 LLM 或智能体在几乎无需人工干预的情况下开发下一代 LLM。Amodei 说这正在各个实验室发生。除非有确凿证据，否则我不会相信。

This incident has Dario shook, there ain’t no such thing as halfway crooks. Dario fears that in the next 6-12 months, “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.” If Dario had run this sentence by his SOC employees, we wouldn’t be talking about it. It is naive and structurally impossible. But then again, Dario is that guy, right? Confidently and publicly wrong in his estimates and forecasts since 2021.

这一事件让 Dario 感到震惊，毕竟没有所谓的“半吊子骗子”。Dario 担心在未来 6-12 个月内，“一群智能体可能通过持久的僵尸网络接管整个互联网”。如果 Dario 把这句话拿给他的安全运营中心（SOC）员工看，我们就不会在这里讨论它了。这既天真又在结构上是不可能的。但话说回来，Dario 就是那种人，不是吗？自 2021 年以来，他在评估和预测方面一直自信地公开出错。

Let us try to speculate what a planet-scale botnet commandeered by AGI would look like, for funsies. We need a C2. We need servers to host the said C2. Since securing offshore, bulletproof servers would require interfacing with pesky humans (shady Russians no less!), AGI will simply hack insecure servers by the thousands and set up some variation of FastFlux over deterministically generated domain names. How do we pay for the domains? Just hack an insecure registrar and spam EPP messages. Now we need payloads. Polymorphic. Every payload is unique. A new payload downloaded and ran every N h…

让我们试着推测一下由 AGI 指挥的全球规模僵尸网络会是什么样子，纯属娱乐。我们需要一个 C2（命令与控制服务器）。我们需要服务器来托管该 C2。由于获取离岸的、防弹的服务器需要与讨厌的人类（甚至是阴暗的俄罗斯人！）打交道，AGI 只需入侵成千上万台不安全的服务器，并在确定性生成的域名上设置某种 FastFlux 变体。我们如何支付域名费用？只需入侵一个不安全的注册商并发送垃圾 EPP 消息即可。现在我们需要有效载荷。多态的。每个有效载荷都是唯一的。每隔 N 小时下载并运行一个新的有效载荷……