Everything is about to “go dark”
Everything is about to “go dark”
一切即将“陷入黑暗”
I’m coming down from spending a few days at Usenix Security, right here in my hometown of Baltimore. This means that my days have been taken up with two kinds of conversation: first, explaining to colleagues why Baltimore isn’t actually like The Wire. And second, trying not to talk about AI. Here I’m going to break that second rule. I have many worries about what AI means for our field, for various definitions of “field”. But in this post I want to focus on just one thing I’ve started worrying about, and it’s a perverse thing: specifically, I’m concerned that AI is going to make software much too secure. While that doesn’t sound so bad on the surface, there’s a consequence to this. I mean something very specific: I’m concerned that U.S. intelligence and law enforcement agencies are about to go dark, meaning: that they’re going to suddenly lose a huge portion of their capability. And that this isn’t going to be simply a problem for those agencies, but also for those of us who value computer security and privacy in general.
我刚结束在我的家乡巴尔的摩举行的 Usenix 安全会议，在那儿待了几天。这意味着我这几天的生活被两类对话占据：首先，向同事解释为什么巴尔的摩实际上并不像美剧《火线》（The Wire）里演的那样；其次，努力不去谈论人工智能。在这里，我要打破第二个规则。对于人工智能对我们领域（基于各种对“领域”的定义）意味着什么，我有很多担忧。但在本文中，我想只关注我开始担心的一件事，而且这是一件反常的事：具体来说，我担心人工智能会让软件变得过于安全。虽然表面上听起来不错，但这会带来一个后果。我的意思是：我担心美国情报和执法机构即将“陷入黑暗”（go dark），这意味着：他们将突然失去很大一部分能力。而且这不仅是这些机构的问题，对于我们这些重视计算机安全和隐私的人来说，也是一个问题。
Going Dark, and the era of law enforcement hacking
“陷入黑暗”与执法黑客时代
To explain how we got here, we need to talk about recent history. This actually gives me a real excuse to reference The Wire, just because it’s a perfect snapshot of what electronic surveillance looked like way back in 2002. If you’ve seen the first season, you’ll recall that it’s about cops wiretapping drug dealers who use payphones and burners. The mobile phones in the show are relatively new technology for the time, but from a technological perspective nothing in this scenario would have shocked a cop who jumped forward from, say, 1989. In less than a decade from the premier, everything in those episodes became totally quaint.
要解释我们是如何走到这一步的，我们需要谈谈近期的历史。这实际上给了我一个引用《火线》的绝佳借口，因为它完美地展现了 2002 年电子监控的样子。如果你看过第一季，你会记得那是关于警察窃听使用公用电话和一次性手机的毒贩的故事。剧中出现的手机在当时是相对较新的技术，但从技术角度来看，如果一个 1989 年的警察穿越到那时，剧中的场景并不会让他感到震惊。在剧集首播不到十年后，那些剧集里的所有东西都变得极其过时了。
The change began in the late 2000s, thanks to the rise of smartphones and texting. Because smartphones can actually store data as well as conveying it, the contents of those phones quickly became a useful new source of law-enforcement capability. Or they were until 2010, when Apple began encrypting iPhone storage using a key derived from the user’s passcode (Android phones followed shortly thereafter.) The next year, Apple deployed end-to-end encryption in iPhone text messages. By 2014, a tiny texting startup named WhatsApp had gathered 600 million users worldwide. By 2016 those users, now nearly a billion strong, were all using default end-to-end encrypted messaging and calls. These two trends — the move from calls to texts, and texts to encrypted data — happened very rapidly.
这种变化始于 21 世纪 00 年代末，这要归功于智能手机和短信的兴起。由于智能手机不仅能传输数据，还能存储数据，手机里的内容很快成为执法部门获取信息的新来源。直到 2010 年，苹果开始使用从用户密码派生的密钥加密 iPhone 存储空间（安卓手机紧随其后），情况才发生改变。次年，苹果在 iPhone 短信中部署了端到端加密。到 2014 年，一家名为 WhatsApp 的小型短信初创公司在全球拥有了 6 亿用户。到 2016 年，这些用户已接近 10 亿，且都在默认使用端到端加密的消息和通话。这两种趋势——从通话转向短信，再从短信转向加密数据——发生得非常迅速。
The FBI and law enforcement agencies were not insensitive to what was happening. In 2014, Director Comey announced an initiative called Going Dark, which would launch a “national conversation” about what providers could do — or be compelled to do — to make these new communications media legible to law enforcement and counterintelligence. In 2016, the agency quit talking and took their theory to court. When a terrorist attack left the FBI holding a shooter’s locked iPhone, the agency ordered Apple to give them access. The company refused.
联邦调查局（FBI）和执法机构并非对正在发生的事情毫无察觉。2014 年，局长科米（Comey）宣布了一项名为“陷入黑暗”（Going Dark）的倡议，旨在发起一场“全国性对话”，讨论服务提供商可以做什么——或被迫做什么——以使这些新的通信媒介能够被执法和反间谍部门读取。2016 年，该机构停止了对话，并将他们的理论诉诸法庭。当一起恐怖袭击导致 FBI 手中握有一部枪手的锁定 iPhone 时，该机构要求苹果公司提供访问权限。苹果公司拒绝了。
What broke the stalemate — and, to some extent, ended “Going Dark” itself — was something that neither the FBI nor Apple expected. An outside company announced that there was no need for Apple’s assistance: they could simply hack the phone. The Apple v. FBI case has turned out to be a microcosm of the whole Going Dark debate. For the next decade, law enforcement and intelligence agencies continued to ask for “exceptional access” backdoors, but the urgency was gone. Agencies and manufacturers both knew that law enforcement could and would purchase targeted hacking tools like GrayKey (for phone unlocking), or even remote exploitation tools like NSO Group’s Pegasus, if they needed them badly enough. Vendors like Apple and Google continued to play a vigorous defense, closing vulnerabilities as soon as they learned about them. But offensive vulnerability hunters consistently managed to keep the edge. And now there’s a very good chance that all this is about to be history.
打破僵局——并在某种程度上终结了“陷入黑暗”本身——的是 FBI 和苹果都没预料到的事情。一家外部公司宣布，根本不需要苹果的协助：他们可以直接破解手机。苹果诉 FBI 案最终成为了整个“陷入黑暗”辩论的缩影。在接下来的十年里，执法和情报机构继续要求获得“特殊访问”后门，但那种紧迫感已经消失了。机构和制造商都知道，如果执法部门有足够强烈的需求，他们可以而且确实会购买像 GrayKey（用于手机解锁）这样的定向黑客工具，甚至是像 NSO Group 的 Pegasus 那样的远程利用工具。像苹果和谷歌这样的供应商继续进行强有力的防御，一旦发现漏洞就立即修复。但进攻性的漏洞猎人始终保持着优势。而现在，这一切很有可能即将成为历史。
The era of AI bug hunting is here
AI 漏洞挖掘时代已经到来
This April (just four months ago!) Anthropic announced a new model called Mythos that happened to be unusually skilled at software vulnerability finding. The U.S. government temporarily blocked its export, restricting access to U.S. agencies and trusted vendors. While the ban was dramatic and made for good PR, it turned out to be mostly pointless. OpenAI, along with Chinese open-weight model labs like Z.ai and Moonshot, have since demonstrated that vulnerability finding isn’t something that a single lab is likely to hold a monopoly on. The list of serious vulnerabilities that these models have found is getting scarier (or more impressive) by the day.
今年四月（就在四个月前！），Anthropic 发布了一款名为 Mythos 的新模型，它在发现软件漏洞方面表现出非凡的技能。美国政府暂时禁止了其出口，限制仅供美国机构和受信任的供应商访问。虽然这项禁令很引人注目，也起到了很好的公关效果，但事实证明它基本毫无意义。此后，OpenAI 以及像 Z.ai 和月之暗面（Moonshot）这样的中国开源权重模型实验室已经证明，漏洞挖掘不太可能被单一实验室垄断。这些模型发现的严重漏洞列表正变得一天比一天更可怕（或更令人印象深刻）。
At first glance, this might seems like good news for the offensive team, and for hackers in general. But I doubt that’s how this will play out in the long term. Defenders are now in the process of patching every bug they can find — often decades worth of bugs — and the backlog feels huge. But they’re making progress. Entire CI toolchains are being rebuilt to incorporate AI-based vulnerability scanning before software ever reaches the point where a human will touch it. While I doubt this means that every bug will be found (even calculating the number of bugs in a piece of code is probably uncomputable), in the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon. Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs. Obviously I think this is great. But for law enforcement and offensive intelligence agencies, it’s going to be a nightmare. For the first time since 2010, law enforcement might experience what it looks like to really “go dark”, across a huge category of advanced (well-maintained) devices and pieces of software. So how is this a problem? The debate…
乍一看，这对于进攻方和黑客来说似乎是个好消息。但我怀疑从长远来看情况会是这样。防御者现在正忙于修补他们能找到的每一个漏洞——通常是几十年的漏洞积累——积压的工作量看起来非常巨大。但他们正在取得进展。整个持续集成（CI）工具链正在被重构，以便在软件到达人类接触点之前就整合基于 AI 的漏洞扫描。虽然我怀疑这意味着每一个漏洞都能被发现（甚至计算一段代码中的漏洞数量可能都是不可计算的），但在现实世界中，我们很可能会在“有用漏洞”的数量上触及某种上限，而且很可能很快就会达到。因此：在接下来的两年里，主要的软件很可能会耗尽可远程利用的漏洞。显然，我认为这很棒。但对于执法和进攻性情报机构来说，这将是一场噩梦。自 2010 年以来，执法部门可能首次体验到在大量先进（维护良好）的设备和软件上真正“陷入黑暗”是什么感觉。那么，这为什么是个问题呢？辩论……