A Decade of Kill Bugs

本文为原文前 6,000 字符的节选翻译,完整内容请查看原文。

A Decade of Kill Bugs

On the morning of September 24th 2026, Endcrawl had a brief outage. I’d just typed: sudo kill -TERM -1490947 And instantly all services went down. Right away I blamed myself. Maybe I mistyped that number! Maybe my code had a signal handling bug. Maybe I don’t understand Unix as well as I think I do. Maybe I’m an idiot!

2026年9月24日上午,Endcrawl 发生了一次短暂的宕机。我当时刚输入了:sudo kill -TERM -1490947,瞬间所有服务都瘫痪了。我立刻责怪自己。也许是我输错了数字!也许我的代码有信号处理错误。也许我并没有自己想象中那么了解 Unix。也许我就是个白痴!

As it turned out, the outage was caused by a bug in kill(1). This is really surprising. The kill program is 50 years old. Its spec is simple and stable. It’s probably exercised a billion times a day. A bug in kill was so surprising that, once I’d found it, I couldn’t stop there. I needed to know how this bug came to be, and what had led the programmers so badly astray.

事实证明,这次宕机是由 kill(1) 中的一个错误引起的。这真的很令人惊讶。kill 程序已经有 50 年的历史了。它的规范简单且稳定。它可能每天被执行十亿次。kill 中的一个错误是如此令人惊讶,以至于一旦我发现了它,我就无法停下来。我需要知道这个错误是如何产生的,以及是什么导致程序员们走上了如此错误的道路。

When I eventually found the root cause, there was an even bigger surprise. It had led to not just one bug, but a whole series of kill bugs that shipped in Ubuntu 14, 16, 18, 22 and 24 LTS. This post documents a decade of kill bugs, then makes some recommendations.

当我最终找到根本原因时,还有一个更大的惊喜。它导致的不仅仅是一个错误,而是一系列在 Ubuntu 14、16、18、22 和 24 LTS 版本中发布的 kill 错误。这篇文章记录了十年来 kill 的错误,并提出了一些建议。

An Outage Not Understood Happens Again. Back to the outage. I scrambled to put the fire out. After that came a surprise second fire. By the end of all this I was pretty demoralized. I was ready to accept that I’d made a dumb mistake and ready to get on with the work day. There was this nagging feeling though…

未被理解的宕机终将重演。回到宕机事件。我手忙脚乱地去灭火。在那之后,又出现了第二次意外的火情。这一切结束后,我感到非常沮丧。我准备接受自己犯了一个愚蠢的错误,并准备继续一天的工作。但心中总有一种挥之不去的念头……

Remember that final scene in Burn After Reading? “What did we learn Palmer?” “I don’t know sir.” “I don’t fking know either. I guess we learned not to do it again.” “Yes sir.” “Although I’m fked if I know what we did…” What had just happened? Was I about to be hit with a third outage? How could I be sure? An outage not understood is an outage that happens again. That nagging feeling I had was a call to understand. With a sigh, I put all my other work on hold and started a proper post mortem.

还记得《阅后即焚》的最后一幕吗?“我们学到了什么,帕尔默?”“我不知道,长官。”“我他妈也不知道。我想我们学到了下次别再这么干了。”“是的,长官。”“虽然我他妈根本不知道我们到底干了什么……”刚才发生了什么?我是不是又要遭遇第三次宕机了?我怎么能确定呢?未被理解的宕机就是会再次发生的宕机。我那种挥之不去的念头是在呼唤我去理解真相。叹了口气,我搁置了所有其他工作,开始进行正式的事后调查。

The Proximate Cause. Let’s examine the command that caused the outage: sudo kill -TERM -1490947. The negative number here is a process group identifier, or pgid for short. A pgid is like a pid, but identifies a group of related processes. By convention, killing a positive number sends a signal to a single process, while killing a negative number signals all processes in that process group.

直接原因。让我们检查一下导致宕机的命令:sudo kill -TERM -1490947。这里的负数是进程组标识符,简称 pgid。pgid 类似于 pid,但它标识的是一组相关的进程。按照惯例,杀死一个正数会向单个进程发送信号,而杀死一个负数则会向该进程组中的所有进程发送信号。

I was attempting to cancel a big Endcrawl 7k test render. A normal person would have killed one of the processes involved and let failures cascade to the others. But killing the whole group simultaneously strikes me as cleaner. Process group signaling isn’t some exotic operation. You use it all the time. Every time you ^C a command, the tty sends SIGINT to the foreground process group.

我当时试图取消一个大型的 Endcrawl 7k 测试渲染。正常人会杀死其中一个进程,让故障级联到其他进程。但同时杀死整个组对我来说更简洁。进程组信号发送并不是什么奇特的操作。你一直在使用它。每次你 ^C 一个命令时,tty 都会向后台进程组发送 SIGINT。

Interactive shells run all but the simplest commands in a new process group, which among other things makes job control possible. Process group signaling is in fact an awesome Unix innovation. I love it so much I added it to daemontools, where svc -+h now avoids orphans in a multiprocess service bounce. At any rate, imagine my surprise when the command didn’t just kill those render processes, but killed all processes on the machine save one!

交互式 shell 会在新的进程组中运行除最简单命令之外的所有命令,这使得作业控制成为可能。进程组信号发送实际上是一项了不起的 Unix 创新。我非常喜欢它,以至于我把它添加到了 daemontools 中,现在 svc -+h 可以避免多进程服务重启时产生孤儿进程。无论如何,想象一下当命令不仅杀死了那些渲染进程,还杀死了机器上除一个进程外的所有进程时,我是多么惊讶!

The Bug. Here’s the relevant snippet from procps 4.0.4, the provider of /bin/kill in Ubuntu 24.04 LTS. Can you spot the bug?

错误所在。这是 Ubuntu 24.04 LTS 中 /bin/kill 的提供者 procps 4.0.4 的相关代码片段。你能发现这个错误吗?

opterr=0; /* suppress errors on -123 */
while ((i = getopt_long(argc, argv, "l::Ls:hVq:", longopts, &optindex)) != -1)
switch (i) {
/* ... */
case '?':
if (!isdigit(optopt)) {
xwarnx(_("invalid argument %c"), optopt);
print_usage(stderr);
} else {
/* Special case for signal digit negative * PIDs */
pid = (long)('0' - optopt);
if (!execute_kill((pid_t) pid, signo, use_sigqueue, sigval))
exitvalue = EXIT_FAILURE;
exit(exitvalue);
}
/* ... */
}

My initial guess was wrong. There’s no underflow or overflow bug with -1490947. Notice we’re in the unrecognized option switch case. Here’s what getopt(3) says about unrecognized options: By default, getopt() prints an error message on standard error, places the erroneous option character in optopt, and returns ’?’ as the function result.

我最初的猜测是错的。-1490947 不存在下溢或上溢错误。注意我们处于无法识别的选项 switch case 中。getopt(3) 对无法识别的选项是这样说的:默认情况下,getopt() 会在标准错误上打印错误消息,将错误的选项字符放入 optopt,并返回 ’?’ 作为函数结果。

Aha. The optopt variable only holds the first character after the dash, not the whole negative pid. When character 1 is subtracted from character 0 we get a pid of -1. What happens when you kill pid -1? Bad things: If pid equals -1, then sig is sent to every process for which the calling process has permission to send signals, except for process 1 (init). This ran as root, so the misparsing of -1490947 as -1 killed every process but init. Bug found, mystery solved.

啊哈。optopt 变量只保存破折号后的第一个字符,而不是整个负 pid。当字符 1 从字符 0 中减去时,我们得到的 pid 是 -1。当你杀死 pid -1 时会发生什么?糟糕的事情:如果 pid 等于 -1,那么信号会被发送给调用进程有权发送信号的每个进程,进程 1 (init) 除外。这是以 root 身份运行的,所以将 -1490947 错误解析为 -1 杀死了除 init 之外的所有进程。错误找到了,谜题解开了。

Wait But Why. Except for me this just raised more questions. Why is the code assuming the unrecognized option is a “signal [sic] digit negative PID”? Why is the single digit negative pid case special, and what handles the general case of a multidigit negative pid? Moreover, how could a bug like this exist in 2026, and in a Ubuntu LTS release no less? Has it always been there? No, that couldn’t be. I’ve killed process groups in the past. And the kill -TERM -12345 thing is an ancient Unix incantation. This used to work dammit!

等等,为什么?除了我之外,这反而引发了更多问题。为什么代码假设无法识别的选项是“单数字负 PID”?为什么单数字负 pid 的情况很特殊,而多数字负 pid 的一般情况又是如何处理的?此外,这样的错误怎么可能在 2026 年,而且还是在 Ubuntu LTS 版本中存在?它一直都在那里吗?不,那不可能。我过去曾杀死过进程组。而且 kill -TERM -12345 是古老的 Unix 咒语。这以前是有效的,该死!

Thinking back, I could vaguely remember some negative pid kills gone awry. Never in an “oops this killed everything in sight” way. More in a “weird, nothing happened” way, or a “poof there goes my shell” way. Minor ways more easily shrugged off as user error. It turns out I wasn’t imagining these. The last decade has been a rough one for negative pid killers. There have been no less than FOUR distinct bugs around negative pids. And they all shipped in Ubuntu LTS versions.

回想起来,我隐约记得一些负 pid 杀死操作出了问题。从来没有出现过“哎呀,这杀死了视线内的一切”的情况。更多的是“奇怪,什么都没发生”或者“砰,我的 shell 没了”的情况。这些小问题更容易被当作用户错误而忽略。事实证明,我并没有想象这些。过去十年对负 pid 杀死操作来说是艰难的。围绕负 pid 至少出现了四个不同的错误。而且它们都出现在 Ubuntu LTS 版本中。

The History. Before I tell the tale of four bugs, let’s go even farther back in Unix history. In the beginning Unix had processes. By 1973 it had the kill(1) program. But it took a decade to evolve process groups as we now know them. It wasn’t until Unix V9 in 1986 that you could kill a process group via negative pid. Near as I can tell, negative pid killing has worked fine in the BSDs all these years. In the early 90s this thing called Linux came along. Although Linux initially borrowed a lot of BSD code for its userland, by the late 90s this approach was in legal jeopardy. Everything had to be rewritten, every

历史。在讲述这四个错误的故事之前,让我们回顾一下更久远的 Unix 历史。起初,Unix 拥有进程。到 1973 年,它有了 kill(1) 程序。但花了十年时间才演化出我们现在所知的进程组。直到 1986 年的 Unix V9,你才能通过负 pid 杀死一个进程组。据我所知,这些年来负 pid 杀死操作在 BSD 中一直运行良好。90 年代初,一个叫 Linux 的东西出现了。虽然 Linux 最初为其用户空间借用了大量的 BSD 代码,但到了 90 年代末,这种方法陷入了法律危机。一切都必须重写,每一个