Four Identical Red Runs: One Was Our Watchdog Working, Three Were Its Corpse
Four Identical Red Runs: One Was Our Watchdog Working, Three Were Its Corpse
四次完全相同的红色运行记录：一次是我们的监控在工作，三次是它的“尸体”
A cron in one of our repositories stopped firing on 21 June and nobody noticed for ten weeks. Its only job was to rebuild this site each morning so that blog posts whose publish date had arrived would actually appear. So for ten weeks, scheduled posts did not publish. The failure mode of a cron is silence, and silence is the one thing no dashboard renders. We built a watchdog for it. The watchdog has four scheduled runs in its history and all four are red. Here is what makes this worth writing down: the first red was the watchdog working exactly as designed, and the other three were it dying before it checked anything. From the outside they are indistinguishable — not merely the same colour, but the same run conclusion and the same step-by-step breakdown, line for line.
我们仓库中的一个定时任务（cron）在 6 月 21 日停止运行，且十周内无人察觉。它唯一的任务是每天早上重建本网站，以确保到达发布日期的博客文章能够正常显示。因此，在长达十周的时间里，预定发布的文章未能上线。定时任务的故障模式是“静默”，而静默恰恰是任何仪表盘都无法呈现的状态。为此，我们构建了一个监控程序（watchdog）。该监控程序的历史记录中有四次定时运行，且全部显示为红色。值得记录的是：第一次红色代表监控程序按预期工作，而随后的三次则是它在执行检查前就已崩溃。从外部看，它们完全无法区分——不仅颜色相同，运行结论和逐行分解步骤也完全一致。
The four runs: 33330840430 2026-08-30T19:25:44Z failure 33442245077 2026-08-31T21:36:52Z failure 33550257729 2026-09-01T19:34:08Z failure 33673601655 2026-09-02T19:29:21Z failure
这四次运行记录如下： 33330840430 2026-08-30T19:25:44Z 失败 33442245077 2026-08-31T21:36:52Z 失败 33550257729 2026-09-01T19:34:08Z 失败 33673601655 2026-09-02T19:29:21Z 失败
Expand any of them and you get the same two lines that matter: step 5 Check every scheduled workflow in both public repos = success step 6 File an issue when a schedule has stopped = failure That is run 1. It is also run 2, run 3 and run 4. We checked all four against the API rather than trusting the screen; they agree to the character.
展开其中任何一条，你都会看到同样关键的两行： 步骤 5 检查两个公共仓库中的所有预定工作流 = 成功 步骤 6 当定时任务停止时提交 Issue = 失败 这是第一次运行的结果。第二次、第三次和第四次运行也是如此。我们没有盲目相信屏幕显示，而是通过 API 核对了所有四次记录；它们在字符层面完全一致。
On 30 August, step 6 was red because the watchdog had found a stopped cron and was filing an issue about it. The issue exists, machine-authored, timestamped 19:26:02Z, and it is correct in every particular: datanika-landing :: .github/workflows/daily-rebuild.yml last ran on a schedule event at 2026-06-21T09:54:45+00:00 — 70.4 days ago. Its cron (0 6 * * *) should have fired within 38h. It is active, so this is NOT the 60-day disable; something else is stopping it.
8 月 30 日，步骤 6 显示为红色，因为监控程序发现了一个停止运行的定时任务，并正在为此提交 Issue。该 Issue 由机器自动创建，时间戳为 19:26:02Z，内容完全准确：datanika-landing :: .github/workflows/daily-rebuild.yml 上次定时运行是在 2026-06-21T09:54:45+00:00，即 70.4 天前。其定时任务 (0 6 * * *) 本应在 38 小时内触发。该任务处于活跃状态，因此这并非“60 天不活动自动禁用”机制，而是其他原因导致了中断。
On 31 August, 1 September and 2 September, step 6 was red because the watchdog had crashed in step 5 and there was nothing to report. It verified nothing on any of those nights. What broke it, and it was not a commit. Nothing in our repository changed between 30 and 31 August. What changed was a repository setting: Dependabot became active on the repo. And when Dependabot is on, GitHub starts listing an extra workflow.
8 月 31 日、9 月 1 日和 9 月 2 日，步骤 6 显示为红色，是因为监控程序在步骤 5 就已崩溃，没有任何内容可报告。那几个晚上它什么都没验证。是什么导致了崩溃？并非代码提交。8 月 30 日到 31 日期间，我们的仓库没有任何变动。改变的是仓库设置：Dependabot 在仓库中被激活了。当 Dependabot 开启时，GitHub 会列出一个额外的工作流。
$ gh api repos/OWNER/REPO/actions/workflows —jq ‘.workflows[].path’ .github/workflows/ci.yml .github/workflows/deploy-pointer.yml … dynamic/dependabot/update-graph
That last one has state: active and a display name of “Dependency Graph”. It is not a file. We did not write it, it is not in the tree, and it is not in any branch. GitHub synthesises it. The workflows API returns it; the contents API has never heard of it. It is not documented as an exception anywhere we could find, and it appears in both of our public repositories.
最后那个工作流的状态为“active”，显示名称为“Dependency Graph”。它不是一个文件。我们没有编写它，它不在代码树中，也不在任何分支里。它是 GitHub 合成出来的。工作流 API 会返回它，但内容 API 对此一无所知。我们在任何地方都找不到关于此例外的文档，而且它出现在我们所有的公共仓库中。
Our watchdog walks every workflow the API returns and reads each one’s YAML off the default branch to extract its cron. check=True turns the 404 into CalledProcessError. Nothing catches it. The script dies mid-collection — and because it collects the repos in order, it died on the first one and never reached the second, which is the repo the watchdog was built to watch.
我们的监控程序会遍历 API 返回的每一个工作流，并从默认分支读取其 YAML 文件以提取定时任务信息。check=True 会将 404 错误转化为 CalledProcessError。没有任何异常捕获机制。脚本在收集过程中中途夭折——由于它是按顺序收集仓库的，它在第一个仓库就崩溃了，从未到达第二个仓库，而那正是监控程序原本要监控的目标。
If you have any tooling that enumerates workflows through the API and then reads their files, run this against your repos now:
gh api repos/OWNER/REPO/actions/workflows --jq '.workflows[] | select(.path | startswith(".github/workflows/") | not) | .path'
Anything it prints will 404 on contents/. dynamic/pages/pages-build-deployment shows up the same way once GitHub Pages is enabled, so this is a family, not a one-off.
如果你有任何通过 API 枚举工作流并读取其文件的工具，请立即在你的仓库中运行以下命令：
gh api repos/OWNER/REPO/actions/workflows --jq '.workflows[] | select(.path | startswith(".github/workflows/") | not) | .path'
它打印出的任何内容在访问内容 API 时都会报 404。一旦启用了 GitHub Pages，dynamic/pages/pages-build-deployment 也会以同样的方式出现，所以这是一个普遍现象，而非个例。
The fix is a whitelist, not a dynamic blacklist — GitHub only ever executes workflows out of .github/workflows/, so anything outside that path cannot be a workflow you own, and whatever prefix GitHub invents next year is handled without a code change. It specifically must not become “swallow the 404”. A 404 on a real .github/workflows/*.yml means a file you were asked to check is unreadable — a token scope, a rename, an API change — and that has to stay fatal. Turning it into “no crons found” would make the watchdog report health from an error.
解决方案是使用白名单，而不是动态黑名单——GitHub 只会执行 .github/workflows/ 目录下的工作流，因此该路径之外的任何内容都不可能是你拥有的工作流，无论 GitHub 明年发明什么前缀，代码都无需更改即可处理。特别要注意的是，绝对不能简单地“吞掉 404 错误”。如果一个真实的 .github/workflows/*.yml 文件报 404，意味着你被要求检查的文件不可读（可能是令牌权限、重命名或 API 变更），这必须保持为致命错误。如果将其处理为“未发现定时任务”，会导致监控程序将错误误报为健康状态。
The part that cost us three nights: The bug above is a fifteen-minute fix. The reason it survived three nights is a design problem, and it is the transferable half. Look at the reporting step. Every path through it ends the same way: Three outcomes — filed a new finding, added to an existing finding, could not run at all — collapsed into one signal. The run had to be red for the first two, because that is how a monitor gets your attention. So the third inherited the same colour, and the watchdog’s own catastrophic failure was camouflaged by its success case.
让我们损失三个夜晚的原因：上述 Bug 只需 15 分钟即可修复。它之所以存在了三个晚上，是因为设计问题，这才是值得借鉴的部分。看看报告步骤，每一条路径的结尾都一样： 三种结果——提交了新发现、追加到现有发现、完全无法运行——被合并成了同一个信号。前两种情况运行结果必须是红色，因为这是监控引起你注意的方式。因此，第三种情况（崩溃）继承了同样的颜色，监控程序自身的灾难性故障被其“成功”的报告机制掩盖了。
We have written before about a green that proves nothing and about alerts that could not fire at all, and about the inverse case, a red that means “I found nothing”. This is a fourth shape and it is the meanest of them, because the signal is not useless — it is genuinely informative, one night in four. It just does not carry which thing it means. A monitor has three states, not two: ran and clean, ran and found something, did not run. If two of those share a colour, the pair that shares it is the pair you will confuse.
我们之前写过关于“毫无意义的绿色”和“无法触发的警报”，以及相反的情况——意味着“我什么都没发现”的红色。这是第四种形态，也是最恶劣的一种，因为信号并非毫无用处——它确实提供了信息（四分之一的夜晚是准确的）。问题在于它没有说明它代表哪种含义。监控程序有三种状态，而非两种：运行且正常、运行且发现问题、未运行。如果其中两种状态共用一种颜色，那么这两种状态就是你最容易混淆的。