How FaultBox helped me solve a storage corruption bug I couldn’t reproduce

FaultBox 如何帮我解决了一个无法复现的存储损坏 Bug

I was testing NodeDB-Lite and PageDB through a real memory-layer application built on top of them. NodeDB-Lite is the embedded form of NodeDB for local-first and in-process workloads, while PageDB is the encrypted page store underneath it. That application was part of the test strategy. I did not want to validate the storage stack only through unit tests, fixtures, and controlled benchmarks. I wanted a real workload to keep using it, stress it, restart it, grow its data, and exercise the boundaries that isolated tests usually miss.

我当时正在通过一个构建在 NodeDB-Lite 和 PageDB 之上的真实内存层应用程序来测试它们。NodeDB-Lite 是 NodeDB 的嵌入式版本，适用于本地优先（local-first）和进程内工作负载，而 PageDB 则是其底层的加密页面存储。该应用程序本身就是测试策略的一部分。我不想仅仅通过单元测试、固件和受控基准测试来验证存储栈。我需要一个真实的工作负载来持续使用它、对其施压、重启它、增加数据量，并测试那些孤立测试通常会遗漏的边界情况。

Then the store became corrupted. The visible symptom was an authenticated-page read failure around an FTS path. A page that should have passed its AEAD authentication check did not. The application restarted, opened the same damaged store, hit the failure again, and fell into a restart loop. The hard part was not proving that the store was corrupt. The hard part was reproducing how it became corrupt. I could not reproduce it inside PageDB. I could not reproduce it through NodeDB-Lite. I could not even make the application produce it on demand. I could use the application normally for a while and eventually see the failure, but I did not have a deterministic sequence that caused it.

随后，存储损坏了。明显的症状是 FTS（全文搜索）路径附近出现了已验证页面读取失败。一个本应通过 AEAD 验证检查的页面未能通过。应用程序重启后，打开同一个损坏的存储，再次遇到故障，并陷入了重启循环。困难之处不在于证明存储已损坏，而在于复现它是如何损坏的。我无法在 PageDB 内部复现它，也无法通过 NodeDB-Lite 复现它，甚至无法让应用程序按需重现该问题。我可以正常使用应用程序一段时间，最终看到故障发生，但我无法找到导致该故障的确定性操作序列。

By the way, I still found bugs along the way. Some were real. Some looked close enough to the corruption path that I thought I had finally found the root cause. I fixed them, rebuilt, ran the tests, and went back to dogfooding. The corruption still came back. At that point, I stopped asking: Which storage bug looks plausible? The real question was: Where does it actually go wrong? I kept testing the wrong shape of failure.

顺便说一下，我在过程中确实发现了一些 Bug。有些是真实的，有些看起来非常接近损坏路径，以至于我以为终于找到了根本原因。我修复了它们，重新构建，运行测试，然后继续进行内部测试（dogfooding）。但损坏问题依然存在。在那一刻，我不再问：“哪个存储 Bug 看起来合理？”真正的问题是：“它到底是在哪里出错的？”我一直在测试错误类型的故障。

My strongest theory was freed-page reuse, or something close to a use-after-free inside the store. It was a reasonable theory. If a page had been released and then reused while another structure still referenced it, a later authenticated read could land on bytes that were valid somewhere else but invalid for the page the reader expected. So I attacked that theory with tests. I stressed allocation and reuse. I pushed the FTS path. I exercised structural checks. I wrote increasingly aggressive single-process cases to catch the store corrupting itself. Nothing reproduced the incident.

我最强烈的假设是已释放页面的重用，或者存储内部类似于“释放后使用”（use-after-free）的问题。这是一个合理的假设。如果一个页面被释放后又被重用，而另一个结构仍然引用它，那么后续的已验证读取可能会落到在其他地方有效、但对于读取者预期的页面而言无效的字节上。因此，我通过测试来验证这一假设。我加大了分配和重用的压力，推动了 FTS 路径，执行了结构检查，并编写了越来越激进的单进程用例来捕捉存储的自我损坏。但没有任何测试能复现该事故。

That should have weakened the theory sooner, but a plausible storage theory is difficult to let go when the only evidence you have is the final corrupted state. The process that wrote the bad page might already be gone. The process that detected it only knew that authentication failed now. The logs could show nearby activity, but they did not preserve the store at the moment the failure was detected. A panic reporter was not enough either, because the corruption arrived as a returned error rather than a panic. I had tests for behavior I understood. What I did not have was the evidence needed to understand this behavior.

这本应让我更早地放弃该假设，但当你手中唯一的证据只是最终的损坏状态时，放弃一个看似合理的存储理论是很困难的。写入错误页面的进程可能已经消失了。检测到错误的进程只知道现在验证失败了。日志可以显示附近的活动，但它们没有在检测到故障的那一刻保存存储状态。崩溃报告器（panic reporter）也不够用，因为损坏是以返回错误的形式出现的，而不是以崩溃（panic）的形式。我有针对我所理解行为的测试，但我缺乏理解这种行为所需的证据。

I needed a black box

我需要一个黑匣子

Then it clicked. I needed a black box. When the application detected corruption or crashed, I needed it to record what it still knew immediately, before another restart, cleanup path, or debugging attempt changed the state. I did not start by designing a public crate. I built the smallest recorder I needed for this investigation. The application would emit a structured capture at the detection site. The capture would keep the error and the surrounding report context, then preserve a copy of the corrupt store so I could inspect the exact artifact that had failed.

突然间，我明白了。我需要一个黑匣子。当应用程序检测到损坏或崩溃时，我需要它立即记录下它所知道的一切，在另一次重启、清理路径或调试尝试改变状态之前。我最初并没有打算设计一个公共库（crate），我只是构建了这次调查所需的最小记录器。应用程序会在检测点发出结构化的捕获信息。该捕获信息会保留错误和周围的报告上下文，然后保存一份损坏存储的副本，以便我检查导致失败的确切产物。

That first version did not have breadcrumbs. It did not have the shared-memory ring, native-crash monitor, or the rest of the public FaultBox feature set. PageDB’s internal FaultBox instrumentation came later too. It was minimal. But it worked. The next time the corruption surfaced, the recorder preserved the damaged store instead of leaving me with another disappearing error. I could run fsck against the captured copy and reproduce the bad state offline. That alone changed the investigation. I was no longer trying to rebuild an incident from memory or force it into the wrong test. I had the store that had actually failed.

第一个版本没有面包屑（breadcrumbs），没有共享内存环、原生崩溃监控器，也没有 FaultBox 公共功能集中的其他部分。PageDB 内部的 FaultBox 插桩也是后来才有的。它非常简陋，但它奏效了。下一次损坏出现时，记录器保存了损坏的存储，而不是让我面对另一个转瞬即逝的错误。我可以对捕获的副本运行 fsck，并在离线状态下复现错误状态。仅这一点就改变了调查的进程。我不再试图凭记忆重建事故，也不再强行将其塞入错误的测试中。我拥有了真正导致失败的存储文件。

Across the incidents, one clue kept appearing: the store was already open. That pointed to the boundary my single-process tests could never reproduce. Two processes had opened the same embedded store and were making allocator decisions without shared ownership. The allocator was not independently corrupting itself inside one process. The ownership model around it was broken. Two processes both believed they owned the store, so a locally valid allocation decision in one process could invalidate what the other process believed. The fix belonged at that boundary: one process, one store, enforced with a single-instance lock and a corrected application lifecycle.

在多次事故中，一个线索不断出现：存储已经被打开了。这指向了我单进程测试永远无法复现的边界。两个进程打开了同一个嵌入式存储，并在没有共享所有权的情况下做出了分配器决策。分配器并没有在一个进程内独立损坏，而是其周围的所有权模型崩溃了。两个进程都认为自己拥有该存储，因此一个进程中局部有效的分配决策可能会使另一个进程所认为的内容失效。修复方案在于该边界：一个进程对应一个存储，通过单实例锁和修正后的应用程序生命周期来强制执行。

I rebuilt a clean store and stressed the application again under that rule. The corruption stopped in this dogfood run. There was another problem one layer up. Corruption discovered after the PageDB handle opened could lose its typed meaning while moving through NodeDB-Lite, collapse into a generic error, and feed the restart loop. The same investigation led me to preserve the typed corruption and add bounded recovery in Lite. FaultBox did not diagnose all of that automatically. It did something more basic and, in this case, more useful: it kept the evidence that let me stop chasing the wrong failure shape.

我重建了一个干净的存储，并在此规则下再次对应用程序施压。在这次内部测试中，损坏问题消失了。在上一层还有一个问题：在 PageDB 句柄打开后发现的损坏，在通过 NodeDB-Lite 传输时可能会丢失其类型含义，坍缩为一个通用错误，并导致重启循环。同样的调查引导我保留了类型化的损坏信息，并在 Lite 中增加了有限恢复功能。FaultBox 并没有自动诊断出所有这些问题。它做了一件更基础、但在这种情况下更有用的事情：它保留了证据，让我不再去追逐错误的故障形态。

From a debugging tool to a public crate

从调试工具到公共库

The minimal recorder had solved a problem I had already spent a long time chasing. That was when I thought: This is good. I should make it better, and I should make it public. The first missing piece was obvious. The captured store showed me what had failed, but the report did not show the operations leading up to it. That pushed me to add the breadcrumb flight recorder. The restart loop exposed another problem. Repeated captures could duplicate large store snapshots. A crash recorder that fills the disk while reporting the same failure has created a second incident. That led to…

这个最小化的记录器解决了我花费很长时间追查的问题。那时我想：这很好。我应该把它做得更好，并将其开源。第一个缺失的部分显而易见：捕获的存储向我展示了什么失败了，但报告没有显示导致失败的操作。这促使我添加了面包屑飞行记录器。重启循环暴露了另一个问题：重复的捕获可能会复制大量的存储快照。一个在报告相同故障时填满磁盘的崩溃记录器，实际上制造了第二次事故。这导致了……