Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug

Tailscale 将数据库损坏追溯至 16 年前的 SQLite WAL 重置漏洞

Blog | Insights | August 12, 2026 博客 | 洞察 | 2026 年 8 月 12 日

How we tracked down a 16-year-old SQLite bug 我们是如何追踪到一个 16 年前的 SQLite 漏洞的

At the end of last year, our uptime was pretty shaky. You can see this trend on our status page, and that instability continued into the new year. Many of these outages were caused by a single bug, deep in SQLite. It took months of intense forensics to track it down. 去年年底，我们的正常运行时间（uptime）表现相当不稳定。你可以在我们的状态页面上看到这一趋势，这种不稳定性一直持续到了新的一年。这些中断中有许多是由 SQLite 深处的一个单一漏洞引起的。我们花费了数月时间进行深入的取证分析，才最终追踪到它。

Now we’re in summer, we’re confident that we’ve found the bug, that we understand it—and more importantly, that we’ve fixed it. 现在已是夏季，我们确信已经找到了这个漏洞，理解了它的成因，更重要的是，我们已经修复了它。

We know our customers expect Tailscale to be a reliable service, and for several months we didn’t live up to that promise. That’s disruptive, and we’re sorry. We’re publishing this blog post to explain what went wrong, how we responded, and how we ultimately helped to uncover a long-standing bug in the heart of the SQLite database. 我们深知客户期望 Tailscale 成为一项可靠的服务，但在过去的几个月里，我们未能兑现这一承诺。这造成了干扰，我们深表歉意。我们发布这篇博文是为了解释出了什么问题、我们是如何应对的，以及我们最终如何协助揭开了 SQLite 数据库核心中一个长期存在的漏洞。

Tailscale’s database architecture Tailscale 的数据库架构

While our clients interact with our control plane as a single public endpoint (controlplane.tailscale.com), internally, our control plane is split into a series of coordination servers (or “shards”). Each tailnet lives on one internal shard at a time, but can migrate seamlessly from one to another. These shards are an internal implementation detail: you don’t know what shard your tailnet is on, and you never need to. 虽然我们的客户端作为单一公共端点（controlplane.tailscale.com）与我们的控制平面交互，但在内部，我们的控制平面被拆分为一系列协调服务器（或称“分片”）。每个 tailnet 在同一时间驻留在一个内部分片上，但可以无缝地从一个分片迁移到另一个分片。这些分片属于内部实现细节：你不需要知道你的 tailnet 位于哪个分片上，也永远不需要知道。

Each shard has an SQLite database that holds all the information about the tailnets on that shard. A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used. 每个分片都有一个 SQLite 数据库，用于保存该分片上所有 tailnet 的信息。单一的 Go 进程独占访问该数据库，并为这些 tailnet 提供控制平面服务。这种单写入器设计正是 SQLite 的预期使用方式。

We’ve used SQLite as our primary database since 2022, and we chose it because it’s well-known, reliable, and widely used. SQLite is “boring technology”—in a good way. Many companies use SQLite in much larger deployments without issue, and we expected the same stress-free usage. 自 2022 年以来，我们一直使用 SQLite 作为主数据库，选择它的原因是它知名、可靠且被广泛使用。SQLite 是一种“无聊的技术”——这是褒义。许多公司在规模大得多的部署中使用 SQLite 且没有出现问题，我们也期待着同样轻松的使用体验。

In our current backup pipeline, we take a complete snapshot of the database every few minutes, then upload the entire SQLite file to an S3 bucket. We’d been running this setup without incident since early 2023. 在我们当前的备份流水线中，我们每隔几分钟就会对数据库进行一次完整快照，然后将整个 SQLite 文件上传到 S3 存储桶中。自 2023 年初以来，我们一直运行此设置且未发生任何事故。

Fast forward to August last year, when a data pipeline that reads those S3 backups reported an error in one of our databases. We ran SQLite’s PRAGMA integrity_check command against the backup, and found it was indeed corrupted. SQLite corruption is possible, but it’s highly unusual and not something you should encounter in normal operation. We repaired the affected database, and investigated the cause, but to no avail. 时间快进到去年 8 月，当时一个读取这些 S3 备份的数据流水线报告了我们其中一个数据库的错误。我们对备份运行了 SQLite 的 PRAGMA integrity_check 命令，发现它确实已损坏。SQLite 损坏是有可能的，但这种情况非常罕见，在正常操作中不应遇到。我们修复了受影响的数据库并调查了原因，但无济于事。

When operating at scale, even rare events can occur with some frequency, so we should have been unsurprised when it happened again—and again, and again, and again. In total, we faced 19 separate instances of database corruption over six months before we finally resolved the underlying bug. 在大规模运营时，即使是罕见的事件也会以一定的频率发生，所以当它再次发生——而且是一次又一次地发生时，我们本不该感到惊讶。在最终解决底层漏洞之前，我们在六个月内总共遇到了 19 起独立的数据库损坏事件。

When you hear the phrase “database corruption”, it’s natural to worry about data loss. Because our control plane only handles configuration data, these databases contain metadata about your tailnet and devices, but never your private encryption keys or network traffic. In the earliest incidents, the recovery process meant a handful of newly added devices or configuration changes didn’t persist, and a small amount of metadata had to be re-entered. 当你听到“数据库损坏”这个词时，担心数据丢失是很自然的。由于我们的控制平面仅处理配置数据，这些数据库包含有关你的 tailnet 和设备的元数据，但绝不包含你的私有加密密钥或网络流量。在最早的几次事件中，恢复过程意味着少数新添加的设备或配置更改未能持久化，且需要重新输入少量元数据。

Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. In the early incidents, that downtime was over an hour, but we gradually sped up the recovery process over subsequent incidents. 每当发生损坏时，我们都必须停止该分片上的控制平面进程，以便修复或恢复数据库。这对该分片上的 tailnet 来说非常痛苦，因为在恢复窗口期间，它们的整个控制平面都会消失。在早期的事件中，停机时间超过一小时，但随着后续事件的发生，我们逐渐加快了恢复过程。

Each tailnet is a mesh network, where devices make peer-to-peer WireGuard® connections to each other. When a device joins the tailnet, it has to get a list of other devices from the control plane before it can establish new connections—so if a device came online during the SQLite downtime, it couldn’t connect. While the database was being repaired, devices already online remained connected to each other, but they couldn’t learn about changes to the network. Those tailnets also temporarily lost access to the web-based admin console and the Tailscale API. 每个 tailnet 都是一个网状网络，设备之间通过 WireGuard® 进行点对点连接。当设备加入 tailnet 时，它必须先从控制平面获取其他设备的列表，然后才能建立新连接——因此，如果设备在 SQLite 停机期间上线，它将无法连接。在数据库修复期间，已经在线的设备可以保持相互连接，但无法获知网络的变化。这些 tailnet 也暂时失去了对基于 Web 的管理控制台和 Tailscale API 的访问权限。

There’s also a broader impact on trust. We post a global incident on our status page even when only a small number of tailnets are affected. Many people saw a status page event for an incident that didn’t affect them. Indeed, the majority of shards and tailnets were never involved in a database corruption incident! Nonetheless, repeated downtime erodes trust, whether or not you’re directly affected. 这还对信任产生了更广泛的影响。即使只有少数 tailnet 受到影响，我们也会在状态页面上发布全球性事件通知。许多人看到了与他们无关的事件的状态页面更新。事实上，大多数分片和 tailnet 从未卷入过数据库损坏事件！尽管如此，反复的停机还是会侵蚀信任，无论你是否直接受到影响。

From the very first instance of corruption, we knew this was a serious threat to our reliability, and we threw a lot of engineering time at the problem—but the fix wasn’t easy. 从第一次损坏事件开始，我们就知道这对我们的可靠性构成了严重威胁，我们投入了大量的工程时间来解决这个问题——但修复并不容易。

Trying to find the fault 尝试寻找故障原因

This bug resisted all our initial attempts to find it. 这个漏洞抵御了我们最初所有的查找尝试。

We looked at recent changes, but there weren’t any that seemed relevant. Nobody had been working on our low-level code that interacts with SQLite, because it had all been written years ago and presented no issues up until that point. We re-reviewed all of that code with a fine-toothed comb to look for previously missed bugs, but we didn’t find anything that would cause the corruption we were seeing. 我们查看了最近的更改，但没有任何更改看起来是相关的。没有人修改过我们与 SQLite 交互的底层代码，因为它都是几年前编写的，并且在那之前没有出现过任何问题。我们用显微镜般细致的态度重新审查了所有代码，以寻找之前遗漏的漏洞，但我们没有发现任何会导致我们所见损坏的原因。

We looked for common factors between corruption incidents, but we couldn’t find any. It wasn’t tied to a single shard, or customer, or tailnet feature, or time of day, or load level. We were at a loss for what might be triggering the behaviour. 我们寻找了损坏事件之间的共同因素，但一无所获。它不与特定的分片、客户、tailnet 功能、时间或负载水平挂钩。我们对于是什么触发了这种行为感到束手无策。

This lack of reliable trigger conditions meant we couldn’t reproduce the bug synthetically. Instead, we had to rely on deploying passive, forensic telemetry in our live environment to catch the corruption red-handed. Gathering live diagnostics for a database issue is the last thing we wanted to do, but we had no choice. 缺乏可靠的触发条件意味着我们无法人工重现该漏洞。相反，我们不得不依赖在生产环境中部署被动的取证遥测技术，以当场抓获损坏过程。收集数据库问题的实时诊断数据是我们最不想做的事情，但我们别无选择。

As an additional complication, the corruption didn’t occur on a regular schedule. Sometimes incidents would be hours apart, other times weeks. This made it difficult to predict progress or plan further work, because we were never sure when we’d get our next diagnostic. 另一个复杂因素是，损坏并非按固定时间表发生。有时事件间隔几个小时，有时则间隔数周。这使得预测进展或规划后续工作变得困难，因为我们永远无法确定何时能获得下一次诊断数据。