Nightly backups with rsync hard links: three snapshots cost 1.4 MB where copies would cost 3.5 MB

查看原文 / View Original 编辑 / Edit

Nightly backups with rsync hard links: three snapshots cost 1.4 MB where copies would cost 3.5 MB

使用 rsync 硬链接进行每日备份：三个快照仅需 1.4 MB，而普通复制则需 3.5 MB

Disclosure first: I sell the five-line core of this post for $4 as snapshot.sh, part of DevToolkit (https://payhip.com/b/aEn8M), so I am not neutral. The flag that does the work is free and has been in rsync for decades, and if you would rather write your own eight lines than pay me, that is a reasonable call. 首先声明：我以 4 美元的价格将本文的核心五行代码作为 snapshot.sh 出售，它是 DevToolkit (https://payhip.com/b/aEn8M) 的一部分，所以我并非中立。实现该功能的核心参数是免费的，并且在 rsync 中已经存在了几十年。如果你宁愿自己写那八行代码也不愿付钱给我，这也是完全合理的选择。

Every measurement below is from one Apple Silicon Mac, run today: /bin/bash 3.2.57, /usr/bin/rsync reporting openrsync: protocol version 29 and rsync version 2.6.9 compatible. Worth naming that up front, because most hard-link backup recipes online assume rsync 3.x. —link-dest is advertised in openrsync’s own usage line and it worked in every case below, but I verified —link-dest on the copy of rsync your Mac actually ships rather than trusting the recipe. 以下所有测量数据均来自一台 Apple Silicon Mac，运行环境为：/bin/bash 3.2.57，/usr/bin/rsync 报告为 openrsync：协议版本 29，兼容 rsync 2.6.9。这一点值得提前说明，因为网上大多数关于硬链接备份的教程都假设使用 rsync 3.x。—link-dest 在 openrsync 的用法说明中就有提及，且在下述所有案例中均有效，但我是在 Mac 自带的 rsync 版本上验证了 —link-dest，而不是盲目相信教程。

The one flag: 核心参数：

# products/devtoolkit/snapshot.sh:14-23
STAMP=$(date +%Y-%m-%d_%H%M)
DEST="$ROOT/$STAMP"
LATEST="$ROOT/latest"

# link from last snapshot for hardlink-based dedup
LINK_ARG=""
if [ -d "$LATEST" ]; then
    LINK_ARG="--link-dest=$LATEST";
fi

rsync -a --delete $LINK_ARG "$SRC/" "$DEST/"
ln -sfn "$DEST" "$LATEST"

—link-dest=DIR tells rsync: before you copy a file, look at DIR, and if the same file is already there unchanged, make the new snapshot’s entry a hard link to it instead of writing the bytes again. LATEST is a symlink to the previous snapshot, which is why the chain works. $LINK_ARG is deliberately unquoted at line 22, so that on the very first run, when no latest exists, it expands to no argument at all rather than to an empty one. —link-dest=DIR 告诉 rsync：在复制文件之前，先查看 DIR 目录。如果同一个文件已经在那里且未发生变化，则将新快照中的条目创建为该文件的硬链接，而不是再次写入数据。LATEST 是指向前一个快照的符号链接，这就是链式备份生效的原因。第 22 行的 $LINK_ARG 特意没有加引号，这样在第一次运行时，由于不存在 latest，它会展开为空，而不是一个空参数。

Three snapshots later, the directory looks like a rotation and the inodes say something more interesting: 三个快照之后，目录看起来像是在轮转，而 inode 的情况则更有趣：

$ ls -la backups
drwxr-xr-x 2026-09-30_1855
drwxr-xr-x 2026-09-30_1856
lrwxr-xr-x latest -> /tmp/bk/backups/2026-09-30_1856

# unchanged in both snapshots
$ stat -f 'ino=%i links=%l' backups/2026-09-30_1855/docs/file1.bin
ino=85983680 links=2
$ stat -f 'ino=%i links=%l' backups/2026-09-30_1856/docs/file1.bin
ino=85983680 links=2

# the one file I rewrote between runs
$ stat -f 'ino=%i links=%l' backups/2026-09-30_1855/docs/file0.bin
ino=85983679 links=1
$ stat -f 'ino=%i links=%l' backups/2026-09-30_1856/docs/file0.bin
ino=85984875 links=1

Same inode number in two snapshots, link count 2. That is the entire mechanism, and it is why the space arithmetic comes out like this. 两个快照中 inode 编号相同，链接计数为 2。这就是整个机制，也是空间计算结果如此的原因。

The space, measured: 空间测量结果：

Source tree: 7 files, 1,200,003 bytes, du -sk reports 1,180 KB. Run 1 copies everything. Then I rewrote exactly one 200,000-byte file and ran it again. 源目录：7 个文件，1,200,003 字节，du -sk 报告为 1,180 KB。第一次运行复制所有内容。然后我重写了其中一个 200,000 字节的文件并再次运行。

$ du -sk backups # after run 1
1180
$ du -sk backups # after run 2, one 200 KB file changed
1376
$ du -sk backups # after run 3, only a text file changed
1380
$ du -sk backups/2026-09-30_1856 # what ONE snapshot reports on its own
1180

So: three full-looking snapshots occupy 1,380 KB. Three naive copies would be 3,540 KB. The second snapshot cost 196 KB of new bytes for a 200 KB changed file, and the third cost 4 KB for a 20-byte edit to one text file. Each snapshot still reads as a complete copy of the folder when you cd into it, which is the point: restoring is cp -R, not a delta replay. 因此：三个看起来完整的快照占用了 1,380 KB。而三次普通复制则需要 3,540 KB。第二个快照为 200 KB 的变更文件额外占用了 196 KB，第三个快照为 20 字节的文本编辑占用了 4 KB。当你进入每个快照目录时，它看起来仍然是一个完整的文件夹副本，这就是重点：恢复时只需使用 cp -R，而不是进行增量重放。

The other thing worth knowing is that this holds because du dedupes by inode within one invocation. du -sk on a single snapshot reports 1,180 KB for it, and reporting three of those side by side would be double counting. The honest number is the one measured across the whole backup root. 另一件值得注意的事是，du 在单次调用中会根据 inode 进行去重。对单个快照运行 du -sk 会报告 1,180 KB，如果将三个快照并排计算则会重复计数。真实的数值是针对整个备份根目录测量的结果。

It also survives deletion, which is the reason to keep snapshots at all. I deleted gone.txt from the source between runs 2 and 3: 它还能在删除操作中幸存，这也是保留快照的原因。我在第 2 次和第 3 次运行之间从源目录删除了 gone.txt：

$ ls backups/2026-09-30_1852 # older snapshot
gone.txt stay.txt
$ ls backups/2026-09-30_1853 # snapshot after the deletion
stay.txt

—delete removes it from the new snapshot only. The old copy is a directory entry pointing at an inode that still has a link, so rm on one snapshot cannot empty it. That is genuinely useful for the “I deleted a folder yesterday” case, and it is the only protection here that I would describe as reliable. —delete 仅从新快照中将其删除。旧副本是一个指向仍有链接的 inode 的目录条目，因此在一个快照上执行 rm 无法清空它。这对于“我昨天删除了一个文件夹”的情况非常有用，也是这里唯一称得上可靠的保护机制。

“keep” is snapshots, not days: “keep”指的是快照数量，而不是天数：

# products/devtoolkit/snapshot.sh:7-9
SRC="${1:?usage: snapshot.sh <source> <backup-root> [keep]}"
ROOT="${2:?usage: snapshot.sh <source> <backup-root> [keep]}"
KEEP="${3:-7}"

# products/devtoolkit/snapshot.sh:26-33
# prune old snapshots beyond KEEP
# portable: awk keeps all but the last KEEP lines (BSD head rejects negative -n counts)
count=0
while IFS= read -r old; do
    rm -rf "$old";
    count=$((count+1))
done < <(ls -1d "$ROOT"/20* 2>/dev/null | awk -v k="$KEEP" '{lines[NR]=$0} END {for (i=1; i<=NR-k; i++) print lines[i]}')
[ $count -gt 0 ] && echo "pruned $count old snapshot(s)"
echo "snapshots kept: $(ls -1d "$ROOT"/20* 2>/dev/null | wc -l | tr -d ' ')"

KEEP defaults to 7 and counts directories, not nights. The README in the pack calls the invocation ./snapshot.sh ~/projects /backups 7 “nightly backup, keep 7 days”, and that is only true if you run it exactly once per day. Run it twice a day and 7 means 3.5 days. The sort is lexical over names, which does equal chronological order for the %Y-%m-%d_%H%M stamp, so the pruning itself picks the right victims: with three snapshots and keep 2 it removed the oldest and left the two newest intact and readable. KEEP 默认为 7，它计算的是目录数量，而不是夜晚。包中的 README 将调用方式描述为 ./snapshot.sh ~/projects /backups 7 “每日备份，保留 7 天”，但这只有在你每天精确运行一次时才成立。如果每天运行两次，7 就意味着 3.5 天。排序是基于名称的词法排序，对于 %Y-%m-%d_%H%M 格式的时间戳来说，这等同于时间顺序，因此清理操作会选中正确的对象：在有三个快照且 keep 为 2 时，它删除了最旧的一个，保留了最新的两个。

Two real problems live in that glob. The stamp is minute-resolution, so two runs in the same minute are one snapshot. DEST is not checked for existence. A second run inside the same minute rsyncs into the directory it just made, with —delete, and overwrites the history it was supposed to add: 这段代码中存在两个实际问题。时间戳的分辨率为分钟，因此在同一分钟内运行两次会合并为一个快照。DEST 没有检查是否存在。在同一分钟内进行的第二次运行会使用 —delete 参数同步到刚刚创建的目录中，从而覆盖了本应添加的历史记录：

$ /bin/bash snapshot.sh src bk # src/state.txt contains ALPHA
[x] snapshot -> /tmp/snap4/bk/2026-09-30_1847
snapshots kept: 1
$ echo BETA > src/state.txt
$ /bin/bash snapshot.sh src bk # same minute
[x] snapshot -> /tmp/snap4/bk/2026-09-30_1847
snapshots kept: 1
$ ls -1d bk/20* | wc -l
1
$ cat bk/2026-09-30_1847/state.txt
BETA

ALPHA is gone. The script printed a success line naming a snapshot directory, and there is only one directory holding the newest state. A nightly cron never hits this, because cron fires once a day. A human who runs it twice because… ALPHA 消失了。脚本打印了成功信息并命名了一个快照目录，但只有一个目录保存了最新状态。每日定时任务 (cron) 不会遇到这个问题，因为 cron 每天只触发一次。如果人类因为某种原因运行了两次……

← 返回新闻列表 / Back to news list