Four bugs my test suite couldn’t catch

我的测试套件无法捕获的四个 Bug

216 passing tests. A feature that was completely broken. Here is the gap between those two facts, and what I changed afterwards. 216 个测试全部通过，但一个功能却完全瘫痪。以下是这两个事实之间的差距，以及我事后所做的改进。

The feature I am building an encrypted messenger. Messages are end to end encrypted, and the server relaying them cannot read anything. That part worked. What I added was offline delivery. If you message someone whose app is closed, the server should hold the message, hand it over when they come back, then delete it. Nothing kept longer than it needs to be. 我正在构建一个加密通讯软件。消息采用端到端加密，中转消息的服务器无法读取任何内容。这部分功能运行正常。我新增的功能是离线投递：如果用户给离线的人发送消息，服务器应暂存该消息，待对方上线后交付，随后将其删除。确保没有任何内容被过度留存。

I wrote it. I wrote tests for it: unit tests for the storage layer, integration tests against a real Postgres, end to end tests over real WebSocket connections. Every one passed. Then I ran it against the deployed build, closed one browser, sent two messages, and reopened. Nothing arrived. 我编写了代码，也编写了测试：针对存储层的单元测试、针对真实 Postgres 数据库的集成测试，以及基于真实 WebSocket 连接的端到端测试。所有测试均已通过。然而，当我将其部署到生产环境，关闭一个浏览器，发送两条消息并重新打开后，却什么也没收到。

Bug 1: a race my tests could not have

Bug 1：测试无法覆盖的竞态条件

The server hands over held messages the instant the connection opens. The client, meanwhile, loads its decryption keys from browser storage, which is asynchronous. So the messages arrived before there was anything to decrypt them with, and were dropped. My tests never saw it because in tests the key loading was effectively instant. The window between “connected” and “ready to decrypt” existed only on a real machine doing real I/O. 服务器在连接建立的瞬间就会交付暂存的消息。与此同时，客户端从浏览器存储中加载解密密钥，这是一个异步过程。因此，消息在解密密钥就绪前就已经到达，导致被丢弃。我的测试从未发现这一点，因为在测试环境中，密钥加载几乎是瞬时的。“已连接”与“准备好解密”之间的时间差，只存在于进行真实 I/O 操作的真实机器上。

// Before: connect, then restore. The gap is where messages die. // 修改前：先连接，后恢复。消息就是在中间的间隙中丢失的。 socket = connect(room) state = await loadSavedState()

// After: restore, then connect. No gap. // 修改后：先恢复，后连接。消除了间隙。 state = await loadSavedState() socket = connect(room)

The lesson: if your test setup completes instantly and production does not, you are not testing the same system. Anywhere your code says await between “we are live” and “we are ready”, something can arrive in between. 经验教训：如果你的测试环境瞬间完成而生产环境不是，那么你测试的就不是同一个系统。在代码中，凡是处于“已上线”和“准备就绪”之间的 await 语句，都有可能在等待期间接收到数据。

Bug 2: acknowledging the wrong event

Bug 2：确认了错误的事件

This one was worse, because it destroyed data. The server deletes a held message once the client confirms it. My client confirmed on arrival. Arrival is not delivery. The message had arrived at the socket, but the app had not decrypted it, had not stored it, had not shown it to anyone. The server deleted it anyway. The message was gone from both sides. The fix was to confirm only after the message had actually been handled, and to leave anything unhandled with the server so it comes again next time. 这个 Bug 更严重，因为它导致了数据丢失。服务器一旦收到客户端的确认就会删除暂存消息。我的客户端在消息“到达”时就进行了确认，但“到达”并不等于“交付”。消息虽然到达了 Socket，但应用尚未解密、存储或展示给用户，服务器却将其删除了。结果消息在双方都消失了。修复方法是：仅在消息被实际处理后才进行确认，未处理的消息则留在服务器上，以便下次重试。

// Before // 修改前 onMessage(m) { show(m); confirm(m.id) } // confirm fires even if show() threw

// After // 修改后 onMessage(m) { if (handled(m)) confirm(m.id) } // unhandled stays on the server

There is one deliberate exception: a message this device can never read, because it belongs to a conversation whose keys are gone, is still confirmed. Asking for it again would not help. The lesson: “received” and “handled” are different events. Only one of them is safe to delete on. If you are building any at-least-once delivery, be precise about which one you are acting on. 有一个刻意的例外：如果设备永远无法读取某条消息（例如对应的会话密钥已丢失），则仍会确认该消息，因为再次请求它也无济于事。经验教训：“接收”和“处理”是不同的事件。只有在“处理”后删除才是安全的。如果你在构建“至少一次交付”机制，请务必明确你是在哪个环节执行操作。

Bug 3: a cleanup path that never ran

Bug 3：从未执行的清理路径

Once messages were flowing again, I looked in the database and found a copy of every message I had sent, including ones delivered instantly while both people were online. The logic was: store every message, delete it when the recipient confirms. But confirmation only happens for messages delivered from storage. A message delivered live goes straight to the other person, so nothing ever confirms it, so nothing ever deletes it. An active conversation was quietly accumulating a server side copy of itself, and only a weekly sweep cleared it. 当消息恢复正常流动后，我查看数据库发现，我发送的每一条消息都有备份，包括双方在线时即时交付的消息。逻辑是：存储所有消息，并在接收方确认后删除。但确认机制只针对从存储中交付的消息。实时交付的消息直接发送给对方，因此永远不会触发确认，也就永远不会被删除。活跃的会话在服务器端悄悄积累了副本，只能靠每周一次的清理任务来清除。

The fix was to store only when nobody is there to receive it. 修复方法是：仅在无人在线接收时才进行存储。

const othersPresent = room.size - 1 > 0 broadcast(room, message) if (!othersPresent) hold(message) // only when it cannot be delivered now

The lesson: a delete that only runs on one code path is not a delete. When you write “we clean this up later”, check that every path reaches the cleanup, not just the one you had in mind. This one also mattered beyond storage. A server holding copies of an entire conversation is a very different privacy claim from one holding a message for a few seconds. 经验教训：只在一条代码路径上运行的删除操作不是真正的删除。当你写下“稍后清理”时，请检查所有路径是否都能触达清理逻辑，而不仅仅是你预想的那一条。这不仅仅关乎存储，服务器持有整个会话的副本，与仅持有几秒钟的消息，在隐私承诺上有着天壤之别。

Bug 4: state that outlived its owner

Bug 4：生命周期长于所有者的状态

Removing a chat deleted the messages but left the encryption keys behind. Re-add the same person and the app cheerfully tried to resume a conversation the other side had thrown away. The two ends no longer agreed on anything, and nothing could be decrypted. The lesson: when you delete a thing, delete everything derived from it. Orphaned state does not sit there harmlessly; it gets picked up later by code that assumes it is still valid. 删除聊天记录时删除了消息，却留下了加密密钥。重新添加同一个人时，应用会愉快地尝试恢复对方已经丢弃的会话。双方不再达成一致，导致无法解密任何内容。经验教训：当你删除一个对象时，请删除所有衍生内容。孤立的状态不会无害地存在，它稍后会被假设其仍然有效的代码所调用。

What I actually changed

我实际做了什么改变

Not “write more tests”. I had plenty, and they were all green while the feature did not work at all. What these bugs had in common is that every one of them lived in the gap between components: between the socket opening and the keys loading, between the server’s idea of delivered and the client’s, between one delivery path and another. Each component behaved correctly in isolation. The system did not. 不是“编写更多测试”。我原本已经有很多测试，且在功能完全失效时它们全部显示通过。这些 Bug 的共同点在于它们都存在于组件之间的间隙中：Socket 打开与密钥加载之间、服务器对“已交付”的定义与客户端的定义之间、不同交付路径之间。每个组件在隔离状态下表现正常，但整个系统却不行。

So the rule I now follow is simple: anything that touches real storage, a real network or a real other machine gets exercised against the deployed build before I call it done. Not the dev server. Not a harness. The thing I actually shipped, with a real second device, doing the thing a user would do. That one test run found four bugs, two of which lost user data. It took about ten minutes. 所以我现在遵循的规则很简单：任何涉及真实存储、真实网络或真实外部机器的功能，在宣布完成前，必须在部署后的构建版本上进行测试。不是开发服务器，也不是测试工具，而是我实际发布的产品，配合一台真实的第二设备，执行用户会做的操作。那一次测试就发现了四个 Bug，其中两个会导致用户数据丢失。整个过程只花了大约十分钟。

The honest footnote: I found these because I went looking. The feature had shipped in the sense that it was written, reviewed, tested and deployed. If I had trusted the green checkmarks, it would have reached testers as a messenger that silently ate messages. Tests tell you the parts work. They are much worse at telling you the whole thing does. 诚实的注脚：我发现这些 Bug 是因为我主动去寻找了。从代码编写、审查、测试和部署的角度来看，该功能已经“发布”了。如果我盲目信任那些绿色的对勾，它就会以一个“静默吞噬消息”的通讯软件形象呈现在测试人员面前。测试能告诉你各个部分运行正常，但它们在告诉你“整个系统运行正常”这件事上表现得很差。