How a Linux Kernel Upgrade Broke MongoDB on an Azure VM — And How We Fixed It
本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。
How a Linux Kernel Upgrade Broke MongoDB on an Azure VM — And How We Fixed It
It was a normal morning. I woke up, grabbed my coffee, and before I’d even properly sat down my phone was already lighting up — alerts, messages, the whole thing. The app was down. Users couldn’t connect, services were throwing errors, and nobody had the faintest idea why. The confusing part? Nobody had touched anything. No deploy the night before, no config change, no code push. I’d left everything green and healthy when I logged off, and I woke up to a dead server. That’s the kind of outage that really gets under your skin, because there’s nothing obvious to point at. So I SSH’d into the VM, half expecting the whole box to be gone, and instead found MongoDB sitting there flat-out refusing to start. That was the beginning of a morning I hadn’t planned for — and the culprit turned out to be something none of us had on our radar: Azure had quietly upgraded the kernel overnight while we were all asleep. This one caught us completely off guard, which is exactly why I wanted to write it down. Here’s the whole story, including the stuff we tried that didn’t work, because honestly the dead ends are half the lesson.
那是一个平常的早晨。我醒来，喝了杯咖啡，还没坐稳，手机就开始响个不停——警报、消息，各种通知接踵而至。应用程序瘫痪了。用户无法连接，服务报错，没人知道原因。最令人困惑的是，根本没人动过任何东西。前一天晚上没有部署，没有配置更改，也没有代码推送。我下线时一切正常，醒来时服务器却挂了。这种故障最让人心烦，因为根本找不到明显的故障点。于是我通过 SSH 登录到虚拟机，本以为整个系统都崩溃了，结果发现 MongoDB 只是拒绝启动。这开启了一个我意料之外的早晨，而罪魁祸首是我们完全没预料到的：Azure 在我们睡觉时悄悄升级了内核。这件事让我们措手不及，这也是我写下这篇文章的原因。以下是完整的故事，包括我们尝试过但无效的方法，因为说实话，这些死胡同也是经验的一部分。
The incident
It started like most outages do — out of nowhere. One of our apps running on an Azure VM just stopped responding. Users couldn’t connect, services were down, and the first thing we did was SSH in and check whether anything was even alive.
sudo systemctl status mongod
MongoDB was in a failed state — Result: exit-code. And the weird part was how fast it was dying. The service would try to start and then fall over within milliseconds, somewhere around 35-48ms. That’s not a database struggling to load data. That’s something rejecting it before it even gets going.
事故经过
故障像大多数事故一样，毫无征兆地发生了。我们在 Azure 虚拟机上运行的一个应用程序停止了响应。用户无法连接，服务中断，我们做的第一件事就是通过 SSH 登录并检查系统是否存活。
sudo systemctl status mongod
MongoDB 处于失败状态——结果：退出代码。奇怪的是它崩溃的速度非常快。服务尝试启动，然后在几毫秒内（大约 35-48 毫秒）就挂掉了。这显然不是数据库加载数据时遇到的困难，而是某种机制在它启动前就拒绝了它。
Hunting for the cause
Logs first, obviously.
sudo cat /var/log/mongodb/mongod.log | tail -50
Useless. The log only showed a clean shutdown from two days ago. Nothing about why it was refusing to start now. So we went to the system journal instead and filtered for fatal messages:
sudo journalctl -u mongod --since "today" --no-pager | grep -i '"s":"F"'
And there it was, the line that made the whole thing click: MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information. A kernel incompatibility. Not corruption, not permissions, not a config mistake. The kernel.
寻找原因
首先看日志，显而易见。
sudo cat /var/log/mongodb/mongod.log | tail -50
没用。日志只显示了两天前的一次正常关闭。没有任何关于为什么现在拒绝启动的信息。于是我们转而查看系统日志，并过滤致命错误消息：
sudo journalctl -u mongod --since "today" --no-pager | grep -i '"s":"F"'
关键的一行出现了，一切真相大白：MongoDB 无法启动：Linux 内核 6.19 及更高版本与此版本的 MongoDB 存在已知的不兼容性。更多信息请参阅 https://jira.mongodb.org/browse/SERVER-121912。是内核不兼容。不是损坏，不是权限问题，也不是配置错误。是内核。
What actually happened
Turns out Azure had auto-upgraded the Linux kernel on the VM overnight during a maintenance window. The box jumped to 7.0.0-1014-azure, which under the hood is based on Linux kernel 6.19+. MongoDB has a known problem with kernel 6.19 and up — something to do with the rseq system call. And rather than risk running in a broken state, MongoDB just refuses to start at all when it detects that kernel. A hard safety block. We confirmed what we were dealing with:
uname -r
# Output: 7.0.0-1014-azure
uname -a
# Linux 7.0.0-1014-azure ... x86_64
And the MongoDB version:
mongod --version
# db version v8.0.29
So: MongoDB 8.0.29 sitting on a kernel it explicitly refuses to run on. That’s the entire bug in one sentence.
究竟发生了什么
原来 Azure 在夜间的维护窗口期间自动升级了虚拟机上的 Linux 内核。系统跳到了 7.0.0-1014-azure，其底层基于 Linux 内核 6.19+。MongoDB 在内核 6.19 及更高版本上存在已知问题——与 rseq 系统调用有关。为了避免在损坏的状态下运行，MongoDB 在检测到该内核时直接拒绝启动。这是一个硬性的安全拦截。我们确认了当前的情况：
uname -r
# 输出: 7.0.0-1014-azure
uname -a
# Linux 7.0.0-1014-azure ... x86_64
以及 MongoDB 版本：
mongod --version
# db version v8.0.29
结论：MongoDB 8.0.29 运行在一个它明确拒绝运行的内核上。这就是整个问题的核心。
The stuff we tried that didn’t work
Before we figured out the kernel was the issue, we burned time on the usual suspects. Leaving these in because this is the part where you learn what the problem isn’t. First we blamed permissions.
sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown -R mongodb:mongodb /var/log/mongodb
Nothing changed. Then we figured it was a stale lock file. Classic MongoDB-won’t-start cause.
sudo rm -f /var/lib/mongodb/mongod.lock
sudo rm -f /tmp/mongodb-27017.sock
Still nothing. Then we thought maybe upgrading MongoDB would fix it. If it’s a known issue, surely a newer version handles it, right?
sudo apt-get update
sudo apt-get install -y mongodb-org
MongoDB upgraded fine, straight to 8.0.32 — and failed with the exact same error. The kernel check is in 8.0.32 too. So no, upgrading doesn’t help; the block is intentional. Then we tried forcing it with —fork.
sudo mongod --config /etc/mongod.conf --fork
This one actually worked for a bit, but only with the old binary. Once we’d upgraded MongoDB, even —fork ran into the kernel check and died. So that was a dead end too, just a slower one. At this point it was clear the problem wasn’t anything on the MongoDB side at all. It was the kernel, full stop. Which meant the fix had to be at the kernel level.
我们尝试过但无效的方法
在确定是内核问题之前，我们浪费时间排查了常见的嫌疑对象。我把这些写出来，是因为你可以从中了解哪些不是问题所在。首先我们怀疑是权限问题。
sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown -R mongodb:mongodb /var/log/mongodb
没有任何改变。然后我们认为是锁文件过期，这是 MongoDB 无法启动的经典原因。
sudo rm -f /var/lib/mongodb/mongod.lock
sudo rm -f /tmp/mongodb-27017.sock
还是不行。接着我们想，也许升级 MongoDB 能解决问题。既然是已知问题，新版本肯定处理了吧？
sudo apt-get update
sudo apt-get install -y mongodb-org
MongoDB 升级很顺利，直接到了 8.0.32——但报错完全一样。8.0.32 里也有内核检查。所以，升级没用；这种拦截是故意的。然后我们尝试用 —fork 强制启动。
sudo mongod --config /etc/mongod.conf --fork
这招在旧二进制文件上确实管用了一会儿。但一旦升级了 MongoDB，即使是 —fork 也会触发内核检查并崩溃。所以这也是死胡同，只是慢了一点。此时很清楚，问题根本不在 MongoDB 那边。完全是内核的问题。这意味着修复必须在内核层面进行。
The real fix: boot into the older kernel
Here’s the thing that saved us — Azure doesn’t throw away the old kernel when it upgrades. It leaves the previous ones installed on disk. The compatible kernel was still right there, just not the one being booted. We listed what was actually installed:
dpkg --list | grep linux-image-.*-azure
rc linux-image-6.17.0-1021-azure # rc = removed, can't use
ii linux-image-6.17.0-1022-azure # ii = installed ✅
ii linux-image-7.0.0-1014-azure # ii = installed, broken ❌
6.17.0-1022-azure was fully installed (ii), and since 6.17 is below 6.19, it was compatible with MongoDB. We just needed to tell the server to boot into it instead of the new broken one. A quick word on GRUB. If you’ve never had a reason to touch this, GRUB (the Grand Unified Bootloader) is the thing that runs the instant your server powers on, before the OS loads. It picks which kernel to boot. Think of it as the startup selector for your Linux box. By default it grabs the newest kernel — which was exactly our problem, because the newest one was the broken 7.0.0-1014-azure. The config lives at: /etc/default/grub
真正的修复：引导至旧内核
救了我们的是这一点——Azure 在升级时不会丢弃旧内核。它会将之前的内核保留在磁盘上。兼容的内核还在那里，只是没有被引导。我们列出了实际安装的内核：
dpkg --list | grep linux-image-.*-azure
rc linux-image-6.17.0-1021-azure # rc = 已移除，无法使用
ii linux-image-6.17.0-1022-azure # ii = 已安装 ✅
ii linux-image-7.0.0-1014-azure # ii = 已安装，损坏 ❌
6.17.0-1022-azure 已完全安装 (ii)，由于 6.17 低于 6.19，它与 MongoDB 兼容。我们只需要告诉服务器引导至该内核，而不是那个损坏的新内核。简单说一下 GRUB。如果你从未接触过它，GRUB（Grand Unified Bootloader）是服务器开机瞬间、操作系统加载前运行的程序。它负责选择引导哪个内核。可以把它看作 Linux 机器的启动选择器。默认情况下，它会抓取最新的内核——这正是我们的问题所在，因为最新的内核是损坏的 7.0.0-1014-azure。配置文件位于：/etc/default/grub
Pointing GRUB at the good kernel
We edited the GRUB config to default to the older compatible kernel, and gave ourselves a boot menu timeout while we were at it:
sudo sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1022-azure"/' /etc/default/grub
sudo sed -i 's/GRUB_TIMEOUT=0/GRUB_TIMEOUT=5/' /etc/default/grub
What those two changes actually do: GRUB_DEFAULT=0 just boots whatever the first (n
将 GRUB 指向正确的内核
我们编辑了 GRUB 配置，将其默认设置为较旧的兼容内核，并顺便设置了启动菜单超时时间：
sudo sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1022-azure"/' /etc/default/grub
sudo sed -i 's/GRUB_TIMEOUT=0/GRUB_TIMEOUT=5/' /etc/default/grub
这两个更改的作用是：GRUB_DEFAULT=0 只是引导第一个（n