Tracking down a Zsh history data loss bug

Tracking down a Zsh history data loss bug 🐞 published 2026-08-09 in tag debug

追踪 Zsh 历史记录数据丢失漏洞 🐞 发布于 2026-08-09，标签：调试

Table of contents

目录

For many years, I sometimes discovered that commands I was sure I had run were no longer present in my Z shell history file (~/.zsh_history). In this article, I will show you how I tracked down the bug. Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the crash’s core dump was the winning strategy!

多年来，我有时会发现自己确信运行过的命令在 Z shell 历史记录文件（~/.zsh_history）中消失了。在本文中，我将向你展示我是如何追踪这个漏洞的。剧透：最终，通过给 Zsh 打补丁使其强制崩溃并分析崩溃产生的核心转储（core dump）是制胜策略！

The good news first: Zsh 5.9.2 (released July 12th, 2026) contains a fix for this issue — open it after reading this investigation to not spoil the fun. Spoiler: link to the upstream fix Zsh fix 53454.

首先是好消息：Zsh 5.9.2（2026 年 7 月 12 日发布）包含了针对此问题的修复——请在阅读完本调查报告后再打开它，以免破坏乐趣。剧透：上游修复链接 Zsh fix 53454。

The symptom

症状

Occasionally, I noticed that commands I knew I executed the day before were not findable in my shell history, meaning pressing Ctrl+R for backward history search yielded no results. Whenever I noticed this, my shell history file contained only very old entries, with years of newer entries missing. The first few times this happened I just restored my shell history from my daily backup and did not bother investigating any further. But the issue kept happening.

偶尔，我注意到前一天执行过的命令在 shell 历史记录中找不到了，这意味着按下 Ctrl+R 进行反向历史搜索时没有任何结果。每当我注意到这一点时，我的 shell 历史记录文件只包含非常旧的条目，而几年来的新条目都不见了。最初几次发生这种情况时，我只是从每日备份中恢复了 shell 历史记录，并没有深入调查。但这个问题一直在发生。

I noticed that there was no visible corruption in the .zsh_history (no non-printable characters or incomplete lines of text), and that the number of lines in the file was not always the same. What was not clear to me was whether it was Zsh itself, or some other program, or perhaps the combination of multiple zsh(1) processes that caused the issue.

我注意到 .zsh_history 中没有明显的损坏（没有不可打印字符或不完整的文本行），而且文件中的行数并不总是相同的。我不清楚是 Zsh 本身、其他程序，还是多个 zsh(1) 进程的组合导致了这个问题。

My Zsh history config

我的 Zsh 历史记录配置

I set the following history-related options in my ~/.zshrc:

我在 ~/.zshrc 中设置了以下与历史记录相关的选项：

# Load 4000 lines of history (for Ctrl+R backward search), but save O(∞)
HISTSIZE=4000
HISTFILE=~/.zsh_history
SAVEHIST=10000000
# Do not save (adjacent) duplicate entries
setopt HIST_IGNORE_DUPS
# Append history entries to `~/.zsh_history` when commands are run.
setopt INC_APPEND_HISTORY
# …but do not share history (enabled by default in NixOS’s /etc/zshrc).
unsetopt SHARE_HISTORY

In practice, this means my shells are separate sessions that all stream their commands into a shared ~/.zsh_history. The history is intentionally not shared, so when I want to access entries that another shell wrote, I explicitly run exec zsh.

实际上，这意味着我的 shell 是独立的会话，它们都将命令流式传输到共享的 ~/.zsh_history 中。历史记录特意设置为不共享，因此当我想要访问另一个 shell 写入的条目时，我会显式运行 exec zsh。

Tracing the behavior

追踪行为

When I asked for help on Mastodon in December 2024 (mostly in the hope that somebody else already encountered and diagnosed this problem), one suggestion I got was to use file system change monitoring mechanisms like inotify or fsevents to find the culprit that truncates (or changes?) the Zsh history file. The next sections walk through the available options on Linux which I tried.

当我在 2024 年 12 月在 Mastodon 上寻求帮助时（主要是希望有人已经遇到并诊断过这个问题），我得到的一个建议是使用像 inotify 或 fsevents 这样的文件系统变更监控机制，来找出截断（或更改？）Zsh 历史记录文件的罪魁祸首。接下来的章节将介绍我在 Linux 上尝试过的可用方案。

inotify

inotify

The inotify(7) Linux kernel subsystem is one of the oldest file system change monitoring APIs available in Linux (released 2005). To get a good understanding of how Zsh modifies the history file, it is not sufficient to monitor just .zsh_history:

inotify(7) Linux 内核子系统是 Linux 中最古老的文件系统变更监控 API 之一（2005 年发布）。为了深入了解 Zsh 如何修改历史记录文件，仅监控 .zsh_history 是不够的：

midna ~ % inotifywait --monitor .zsh_history
Setting up watches.
Watches established.
.zsh_history OPEN
.zsh_history ACCESS
.zsh_history ACCESS
[…]
.zsh_history ACCESS
.zsh_history CLOSE_NOWRITE,CLOSE
.zsh_history ATTRIB
.zsh_history CLOSE_WRITE,CLOSE
.zsh_history DELETE_SELF
^C

The file is opened, accessed (= read) and then… deleted?! By monitoring the containing directory, we see the whole picture:

文件被打开、访问（= 读取），然后……被删除了？！通过监控所在的目录，我们可以看到全貌：

midna ~ % inotifywait --monitor ~
/home/michael/ OPEN .zsh_history
/home/michael/ ACCESS .zsh_history
/home/michael/ ACCESS .zsh_history
[…]
/home/michael/ ACCESS .zsh_history
/home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history
/home/michael/ OPEN .zsh_history
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history
/home/michael/ OPEN .zsh_history
/home/michael/ ACCESS .zsh_history
/home/michael/ CLOSE_NOWRITE,CLOSE .zsh_history
/home/michael/ CREATE .zsh_history.new
/home/michael/ OPEN .zsh_history.new
/home/michael/ ATTRIB .zsh_history.new
/home/michael/ MODIFY .zsh_history.new
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history.new
/home/michael/ MOVED_FROM .zsh_history.new
/home/michael/ MOVED_TO .zsh_history
/home/michael/ CLOSE_WRITE,CLOSE .zsh_history

So Zsh reads the old history file contents, writes them to a new file, then renames the new file over the old one, thereby deleting the old one. Now it makes sense! Unfortunately, we do not see the process IDs (PIDs) of the responsible process for the file system event, not even with the sibling utility fsnotifywait(1), which uses fanotify(7), an API that does provide this information! I checked, and the kernel does send the PID, but fsnotifywait does not display the PID.

所以 Zsh 读取旧的历史记录文件内容，将其写入一个新文件，然后将新文件重命名覆盖旧文件，从而删除了旧文件。现在说得通了！不幸的是，我们看不到负责该文件系统事件的进程 ID (PID)，即使使用使用 fanotify(7) 的同类工具 fsnotifywait(1) 也看不到，尽管该 API 确实提供了此信息！我检查过，内核确实发送了 PID，但 fsnotifywait 没有显示它。

fatrace

fatrace

Luckily, there is fatrace(8), which does display the process name and PID. Here is what Zsh’s history rewriting looks like with fatrace(8):

幸运的是，有 fatrace(8)，它确实会显示进程名称和 PID。以下是使用 fatrace(8) 查看 Zsh 重写历史记录时的样子：

zsh(197994): CWO /home/michael/.zsh_history
zsh(197994): O /home/michael/.zsh_history
zsh(197994): R /home/michael/.zsh_history
zsh(197994): R /home/michael/.zsh_history
[…]
zsh(197994): R /home/michael/.zsh_history
zsh(197994): C /home/michael/.zsh_history
zsh(197994): + /home/michael
zsh(197994): O /home/michael/.zsh_history.new
zsh(197994): W /home/michael/.zsh_history.new
zsh(197994): W /home/michael/.zsh_history.new
zsh(197994): W /home/michael/.zsh_history.new
[…]
zsh(197994): W /home/michael/.zsh_history.new
zsh(197994): CW /home/michael/.zsh_history.new
zsh(197994): <> /home/michael
zsh(197994): CW (deleted)
zsh(197994): C /nix/store/80vwnjjgcrbp41pk927r8lzybjhy0k73-zsh-5.9.1/bin/zsh
[…]

This gives us the PID, so now we can verify whether multiple processes were involved in corrupting the shell history. But, we don’t have any insight into how much data each Zsh PID is reading/writing, so even with a fatrace log, it would still not be clear what happened.

这给了我们 PID，所以现在我们可以验证是否有多个进程参与了 shell 历史记录的损坏。但是，我们无法洞察每个 Zsh PID 正在读取/写入多少数据，因此即使有了 fatrace 日志，仍然不清楚发生了什么。

strace

strace

Of course, one could use strace(1), in particular with its -k flag, to further look into Zsh behavior, but it seems like a logistical nightmare to arrange for every (interactive) Zsh process to get a corresponding strace run, and I was not sure if always-stracing a shell changes behavior in subtle ways, so I did not pursue the strace route. (Once I had a reproducer, s…)

当然，人们可以使用 strace(1)，特别是配合其 -k 标志，来进一步研究 Zsh 的行为，但安排每个（交互式）Zsh 进程都运行相应的 strace 似乎是一场后勤噩梦，而且我不确定一直对 shell 进行 strace 是否会以微妙的方式改变其行为，所以我没有采用 strace 路线。（一旦我有了复现步骤，s…）