本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

The Decoherence Trap: When AI slop drowns out the signal in an endless sea of noise

TL;DR This essay introduces the concept of decoherence: the accumulation of locally valid but mutually incompatible or redundant models within a system, such that the cost of change is dominated by translating among those models and preventing unintended side-effects.

简而言之，本文介绍了“退相干”（decoherence）的概念：即在一个系统中，局部有效但相互不兼容或冗余的模型不断累积，导致变更成本主要被消耗在这些模型之间的转换以及防止意外副作用上。

In other words, it’s what happens when slop drowns out the signal in an endless sea of noise. Decoherence grows relative to the rate of change of a system, and it increases faster the more decoherent a system is, resulting in runaway exponential growth.

换句话说，这就是当垃圾信息在无尽的噪音海洋中淹没了有效信号时所发生的情况。退相干随着系统变化率的增加而增长，且系统越是退相干，其增长速度就越快，从而导致失控的指数级增长。

It is not a new phenomenon, and humans have always countered it through carefully finding ways to do more with less, unifying disparate patterns and models into a cohesive whole. However, the widespread use of AI is currently producing it at a rate that is unsustainable.

这并非什么新现象，人类一直以来都在通过寻找“以少做多”的方法来应对它，将零散的模式和模型统一为一个有机的整体。然而，人工智能的广泛应用目前正以一种不可持续的速度制造这种现象。

The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it.

好消息是，这排除了递归式自我改进失控的阴影——即在任何人反应过来之前就呈抛物线式增长的“智能爆炸”——因为退相干既阻止了人工智能在封闭环境中的自我完善，也限制了它在外部环境中的改进速度。

The bad news is that we are barreling toward a different kind of disaster: Decoherence Collapse. Imagine a building, let’s say a small office tower. When this building was built, they used a type of pipe that will degrade over time and eventually need to be replaced. This is standard infrastructural debt; basic wear and tear, as they say.

坏消息是，我们正冲向另一种灾难：退相干崩溃。想象一座建筑，比如一座小型办公楼。在建造时，使用的是一种会随时间推移而退化并最终需要更换的管道。这是标准的基建债务；也就是人们常说的基本磨损。

Then, let’s say that when it comes time to replace the pipes, you need to change them over to a new standard. This could be because of new building code requirements, because the old pipes simply aren’t made anymore, or because the new pipes are much better and easier to maintain. This is standard technical advancement debt, the need to upgrade and transition due to the forward progress of technology over time.

假设到了更换管道的时候，你需要将其更换为新的标准。这可能是因为新的建筑规范要求，因为旧管道已不再生产，或者因为新管道更好且更易于维护。这是标准的技术进步债务，即由于技术的不断进步而产生的升级和转型需求。

There is a third form of debt that can arise based on how the transition from old pipes to new pipes is done. Consider the following options: Shut down the building and replace all of the pipes at once. This is a very disruptive and expensive method, since you can’t use the building while the process is underway and you have to pay for the replacement up front. But the process is complete by the end of it.

基于旧管道到新管道的转换方式，还会产生第三种债务。考虑以下方案：关闭大楼并一次性更换所有管道。这是一种极具破坏性且昂贵的方法，因为在施工期间你无法使用大楼，且必须预付更换费用。但施工结束后，整个过程也就完成了。

Shut down each floor, one at a time, and replace the pipes on that floor. This is less disruptive overall, but somewhat more expensive because you still have to pay up front, and you have to pay a premium for the extra labor required to keep things running smoothly as the work goes on.

逐层关闭并更换该楼层的管道。这在总体上破坏性较小，但成本略高，因为你仍然需要预付费用，并且为了在施工期间保持大楼正常运行，你必须为额外的劳动力支付溢价。

Replace the pipes as needed, when they begin to fail. This is the least disruptive strategy, and it allows you to amortize the cost of replacing all of the pipes over time, with one small repair after another. For most people, option 3 is very appealing. It allows them to keep things running smoothly and to spread out the cost of maintenance.

在管道开始损坏时按需更换。这是破坏性最小的策略，它允许你将更换所有管道的成本分摊到一段时间内，通过一次次小规模维修来完成。对大多数人来说，方案三非常有吸引力。它能让一切保持平稳运行，并分散维护成本。

And if we can just make like the Ship of Theseus[1] and replace the pipes one at a time, why shouldn’t we? Well, for one thing, it creates decoherence. And decoherence is, by far, the worst form of tech debt.

如果我们能像“忒修斯之船”那样逐一更换管道，为什么不这样做呢？嗯，首先，这会产生退相干。而退相干，绝对是技术债务中最糟糕的一种。

Evolution The Ship of Theseus actually gives us a good lens for understanding the true scope of decoherence. We know how to build ships made of wood, and we know how to build ships made of steel. You can repair a ship one piece at a time with the same material, as the original thought experiment goes, and the ship remains functional, with all of the original designs and structures that went into it intact.

进化：忒修斯之船实际上为我们理解退相干的真正范畴提供了一个很好的视角。我们知道如何建造木船，也知道如何建造钢船。正如最初的思想实验所言，你可以用相同的材料逐件修理船只，船只依然保持功能，且其原有的设计和结构完好无损。

But do we know how to build every iteration of a ship in between wood and steel? Could you, say, slowly replace every piece of a man-of-war and incrementally turn it into a modern destroyer? And have it functional (for some definition of functional) at every step in the process?

但我们知道如何建造木船与钢船之间每一个迭代版本的船只吗？比方说，你能否缓慢地更换一艘战舰的每一个部件，并将其逐步变成一艘现代驱逐舰？并且在过程中的每一步都保持其功能性（基于某种功能定义）？

It’s not that the answer is an absolute “no”. If we are creative enough, if we had the best minds working on it, then I’m sure there would be some path to doing this. But it would require ingenuity and skill and craftsmanship. It would require attention to detail. And, crucially, you would really want to complete the process, because the half-man-of-war, half-destroyer vessel is likely not going to be very good in modern combat.

答案并非绝对的“不”。如果我们足够有创造力，如果我们有最优秀的人才参与其中，我相信一定会有某种方法可以做到。但这需要独创性、技巧和工艺。它需要对细节的关注。而且至关重要的是，你必须真正想要完成这个过程，因为那种半战舰、半驱逐舰的船只在现代战斗中很可能表现不佳。

Returning to our plumbing example, we can see how the same strategy plays out regularly in the world of construction. Every time a pipe breaks, the plumber is called, walls are opened, the repair is hand-crafted for that specific area, and then everything is closed back up. If you’re lucky, the plumber will leave detailed notes, and if you’re not, then years later some poor sod is going to open that wall back up and say to themselves, “What the hell is going on here?”

回到我们的管道示例，我们可以看到同样的策略在建筑领域是如何频繁上演的。每当管道破裂，水管工就会被叫来，墙壁被拆开，针对该特定区域进行手工修复，然后一切又被封回去。如果你运气好，水管工会留下详细的记录；如果你运气不好，那么多年后，某个倒霉蛋打开那堵墙时，会对自己说：“这到底是怎么回事？”

Now, multiply that process over time as we slowly replace the pipes in our building, and the end result is a patchwork of old and new pipe that has much more complexity than either system on its own. Every joint, every hand-crafted patch, is its own little bundle of unique forces and properties that are understudied, and thus requires an expert in both systems to fully understand and work with.

现在，随着我们缓慢更换大楼管道，将这个过程随时间推移不断累积，最终的结果是一个新旧管道交织的拼凑体，其复杂程度远超任何单一系统。每一个接头、每一个手工修补处，都是一堆未经充分研究的独特受力和属性，因此需要同时精通两种系统的专家才能完全理解并进行操作。

On the surface, this is fine. The pipes work, why does it matter how they work? As long as it’s a good patch, who cares? Well, what happens if we, say, want to build an annex? For the purposes of this thought experiment, we’ll say that the water for the annex needs to be routed through at least part of our existing building, using the existing pipes. And, crucially, we need to increase the overall water pressure in the system. Now, the question is: Is it safe to increase the pressure?

从表面上看，这没问题。管道能用，它们是怎么工作的又有什么关系呢？只要修补得好，谁在乎呢？那么，假设我们想建一个附属建筑会怎样？为了这个思想实验，我们假设附属建筑的用水需要通过现有建筑的部分管道进行输送。而且至关重要的是，我们需要增加系统中的整体水压。现在的问题是：增加压力安全吗？

If we had all type A or all type B pipes, even ones that had been patched and repaired over time, then this would be a straightforward question. Both systems are well understood, both have tools and processes for calculating their capacity, and people run these calculations all of the time. But, if we have

如果我们全是A型或全是B型管道，即使是那些经过长期修补和维护的管道，这也会是一个简单的问题。两种系统都为人所熟知，都有计算其容量的工具和流程，人们也一直在进行这些计算。但是，如果我们拥有的是……