One fallen power line exposed a growing AI data center problem. Here’s how to fix it.
One fallen power line exposed a growing AI data center problem. Here’s how to fix it.
一条输电线的倒塌暴露了人工智能数据中心日益严重的问题。以下是解决方案。
A power line went down outside of Washington, DC, this week. Normally, the grid would only need a few seconds to recover from such an event. But this one took more than 10 minutes because more than 3 gigawatts of data centers stopped drawing power nearly simultaneously. 本周，华盛顿特区外的一条输电线倒塌。通常情况下，电网只需几秒钟就能从此类事件中恢复。但这次却耗时超过 10 分钟，因为超过 3 吉瓦（GW）的数据中心几乎同时停止了电力消耗。
The event caused voltage across the PJM grid to spike from Northern Virginia to Chicago, according to data collected by Ting Labs, a startup that runs an IoT sensor network out of people’s electrical sockets. The event didn’t cause a blackout, but it did cause lights across the region to flicker. 根据物联网传感器网络初创公司 Ting Labs（通过用户插座收集数据）的数据，该事件导致从北弗吉尼亚州到芝加哥的 PJM 电网电压出现激增。虽然此次事件并未导致停电，但确实造成了该地区各处的灯光闪烁。
The incident demonstrated the effect that data centers can have on the grid — an outcome that experts believe will become more frequent. Northern Virginia, which is in PJM’s territory, is home to the highest concentration of data centers in the world. “It’s the canary in the coal mine,” Ricardo de Azevedo, CTO at ON.Energy, told TechCrunch. These sorts of events involving large loads like data centers are “happening more and more,” he added. 此次事故展示了数据中心对电网的影响——专家认为这种情况未来会更加频繁。位于 PJM 辖区内的北弗吉尼亚州拥有全球最密集的数据中心。ON.Energy 的首席技术官 Ricardo de Azevedo 对 TechCrunch 表示：“这就像煤矿里的金丝雀（预警信号）。”他补充说，这类涉及数据中心等大负荷的事件正“越来越多地发生”。
The event echoes one that happened two years ago, also on PJM’s grid, and it could foreshadow larger events if data centers aren’t built to more elegantly handle disruptions to power supplies. The PJM Interconnection manages grids from New Jersey to Illinois and serves 67 million customers, making it the largest grid operator in the United States. 该事件与两年前发生在 PJM 电网上的另一起事件如出一辙。如果数据中心在设计上不能更从容地应对电力供应中断，这可能预示着未来会发生更严重的事故。PJM Interconnection 管理着从新泽西州到伊利诺伊州的电网，服务 6700 万用户，是美国最大的电网运营商。
When the power line went down this week, it triggered data centers to switch to backup power, and about 3.1 gigawatts of load vanished in about 30 seconds, according to PJM data. The grid appeared to recover somewhat, but a short time later additional loads dropped off. At its peak, PJM’s grid had an extra 3.49 gigawatts of electricity on it. It took another 11 minutes before it stabilized. 根据 PJM 的数据，本周输电线倒塌时，触发了数据中心切换至备用电源，约 3.1 吉瓦的负荷在 30 秒内消失。电网似乎有所恢复，但不久后又有额外的负荷掉线。在峰值时，PJM 电网上出现了 3.49 吉瓦的电力盈余。电网在 11 分钟后才恢复稳定。
The disconnected data centers represented around 3% of total demand on PJM at the time, according to Reuters. A few percent may not sound like much, but the electrical grid needs to operate in a state of near-perfect balance, with supply and demand closely matched. If they don’t, voltages can sag or spike. 据路透社报道，断开连接的数据中心约占当时 PJM 总需求的 3%。几个百分点听起来可能不多，但电网需要在近乎完美的平衡状态下运行，供需必须紧密匹配。否则，电压就会出现下降或激增。
The grid and devices connected to it can tolerate small fluctuations, but if those fluctuations grow too large, they’ll trigger failsafes within the grid or within individual facilities, causing them to disconnect. When data centers in Northern Virginia sensed the fluctuation caused by the failed power line, they switched to backup power, which removed their load from the grid. 电网和连接的设备可以容忍小幅波动，但如果波动过大，就会触发电网或设施内部的故障保护机制，导致其断开连接。当北弗吉尼亚州的数据中心感知到输电线故障引起的波动时，它们切换到了备用电源，从而从电网中移除了负荷。
As more data centers made the switch, they removed greater amounts of load from the grid. What started as a relatively small drop in supply became an even larger drop in demand, sending supply surging and causing light bulbs to flicker. 随着更多数据中心进行切换，它们从电网中移除了更多的负荷。最初相对较小的电力供应下降，演变成了更大的需求下降，导致电力供应激增，进而引发了灯泡闪烁。
Most data centers make decisions in a split second, and those that disconnected this week appear to be no different. When the voltage dip reached them, they all decided to disconnect within a few seconds of each other, Ali Zain Banatwala, senior market models specialist at the Independent Electricity System Operator, told TechCrunch. “We need to figure a way for these loads that are located next to each other to sequentially either disconnect or reconnect,” he said. 大多数数据中心在瞬间做出决策，本周断开连接的数据中心也不例外。独立电力系统运营商（IESO）的高级市场模型专家 Ali Zain Banatwala 对 TechCrunch 表示，当电压骤降波及它们时，它们都在几秒钟内决定断开连接。“我们需要找到一种方法，让这些相邻的负荷能够按顺序断开或重新连接，”他说。
A more orderly process would allow grid operators to develop more robust procedures in advance. Alternatively, data centers could be built to absorb disruptions and not turn their backs to them. One startup, ON.Energy, has been working on a product to help data centers — and the grid — ride through events like the one that occurred this week. 更规范的流程将使电网运营商能够提前制定更稳健的程序。或者，数据中心在建设时应具备吸收干扰的能力，而不是直接切断连接。初创公司 ON.Energy 一直在开发一种产品，旨在帮助数据中心和电网平稳度过类似本周发生的事件。
The company has developed an uninterruptible power supply for an entire data center campus, covering not just servers but also chillers and other equipment. The company essentially hides the data center behind a bank of batteries connected to sophisticated power conversion equipment. All the grid “sees” is one consistent, well-behaved load rather than the peaks and valleys from each individual part of the data center. 该公司为整个数据中心园区开发了不间断电源系统，不仅覆盖服务器，还包括冷却器和其他设备。该公司本质上是将数据中心隐藏在一组连接着精密电力转换设备的电池组之后。电网“看到”的只是一个稳定、表现良好的负荷，而不是数据中心各个部分产生的峰谷波动。
ON.Energy’s system allows data centers to ramp computing workloads up and down, including AI training, without bothering the grid. Perhaps more important, it also means that data centers can absorb power fluctuations from the grid. Rather than disconnecting from the grid, ON.Energy’s system can use any extra power to charge its batteries, and if the flow dips, the system can dispatch power to servers. ON.Energy 的系统允许数据中心在不干扰电网的情况下调整计算工作负载，包括人工智能训练。更重要的是，这意味着数据中心可以吸收来自电网的电力波动。ON.Energy 的系统不会断开电网连接，而是利用多余的电力为电池充电；如果电力供应下降，系统则会向服务器输送电力。
Plus, it can follow the grid’s lead within milliseconds, preventing sags or surges like the ones that caused this week’s problem for PJM. ON.Energy is currently installing a total of 3 gigawatts worth of its systems at four different data center campuses, de Azevedo said. 此外，它能在几毫秒内响应电网需求，防止出现本周导致 PJM 问题的电压骤降或激增。de Azevedo 表示，ON.Energy 目前正在四个不同的数据中心园区安装总计 3 吉瓦容量的系统。
Grid managers have also woken up to the problem. ERCOT, for example, is going to require large loads like data centers to “ride through” disruptions, de Azevedo said. The clock is ticking, though. The mass disconnection this week was twice as large as a similar event in 2024, when 60 data centers simultaneously disconnected, pulling 1.5 gigawatts of load from the grid. 电网管理者也已经意识到了这个问题。de Azevedo 说，例如，德克萨斯州电力可靠性委员会（ERCOT）将要求数据中心等大负荷用户具备“穿越（ride-through）”干扰的能力。然而，时间紧迫。本周的大规模断开事件规模是 2024 年类似事件的两倍，当时有 60 个数据中心同时断开，从电网中移除了 1.5 吉瓦的负荷。
Back then, data centers accounted for about 6% of PJM’s load, according to Synapse Energy Economics. By 2040, they are expected to make up 24%. If the problem isn’t addressed soon, things could get a lot worse. 据 Synapse Energy Economics 统计，当时数据中心约占 PJM 负荷的 6%。预计到 2040 年，这一比例将达到 24%。如果这个问题不能尽快解决，情况可能会变得更糟。