Webhook Retries Aren’t Optional: The 6-Hour Silent Failure That Changed How We Build Them

Webhook 重试并非可选项：那次 6 小时的静默故障如何改变了我们的构建方式

We found out our webhook delivery was broken the same way most teams do: a customer told us. Not an alert, not a dashboard going red — an email asking why a payment status update from three hours earlier had never shown up in their system. By the time we’d pulled the logs, we counted just over six hours where a downstream endpoint had been failing silently, and every event meant for it was gone. Not queued, not retried. Gone. 我们发现 Webhook 发送功能故障的方式和大多数团队一样：客户告诉我们的。没有告警，仪表盘也没有变红，只有一封邮件询问为什么三小时前的支付状态更新一直没有出现在他们的系统中。当我们调取日志时，发现下游端点已经静默失败了六个多小时，所有发往该端点的事件都丢失了。没有进入队列，没有重试。直接消失了。

The endpoint in question belonged to a mid-sized integration partner whose ops team had rotated a TLS cert and, in the process, briefly served a handshake our client library didn’t like. Our webhook sender treated that as a delivery failure, logged it at a level nobody was watching, and moved on. There was no retry logic — we’d built a fire-and-forget publisher because in two years of running it, deliveries had basically always succeeded. That streak was the problem. It meant we’d never had to think about what “basically always” leaves out. 出问题的端点属于一家中型集成合作伙伴，他们的运维团队在轮换 TLS 证书时，导致握手过程出现了我们的客户端库无法识别的情况。我们的 Webhook 发送器将其视为发送失败，记录在一个无人关注的日志级别中，然后就继续处理后续任务了。当时没有任何重试逻辑——我们构建的是一个“即发即忘”（fire-and-forget）的发布者，因为在运行的两年里，发送基本上总是成功的。这种“连胜”恰恰是问题所在。这意味着我们从未思考过“基本上总是”之外的情况。

The fix looked simple on a whiteboard and took us about three weeks to get right in production, mostly because the interesting failure modes only show up under real load and real partner flakiness. 这个修复方案在白板上看起来很简单，但我们在生产环境中花了大约三周时间才真正搞定，主要是因为那些有趣的故障模式只有在真实的负载和合作伙伴不稳定的情况下才会显现。

Exponential backoff, not fixed intervals

指数退避，而非固定间隔

Our first instinct was to retry every 30 seconds for five minutes. That’s exactly the pattern that turns one struggling downstream service into a thundering herd — every failed webhook from every affected customer retries in lockstep, which is a great way to keep a recovering endpoint down. We moved to backoff starting at 10 seconds and doubling up to a ceiling of about 30 minutes, with jitter of ±20% on every attempt so retries from different events don’t stack on the same tick. Six attempts over roughly two hours, then the event moves to a dead-letter queue instead of disappearing. 我们的第一直觉是每 30 秒重试一次，持续五分钟。这正是那种会将一个挣扎中的下游服务变成“惊群效应”（thundering herd）的模式——来自受影响客户的所有失败 Webhook 会同步重试，这简直是让正在恢复的端点彻底崩溃的最佳方式。我们改用了从 10 秒开始的退避策略，每次翻倍，上限约为 30 分钟，并在每次尝试时加入 ±20% 的抖动（jitter），这样不同事件的重试就不会堆积在同一时间点。大约两小时内进行六次尝试，之后事件会进入死信队列（DLQ）而不是直接消失。

The dead-letter queue is the actual fix, not a nice-to-have

死信队列是真正的修复方案，而非锦上添花

The bug wasn’t “we don’t retry enough times” — it was “when retries are exhausted, the event vanishes.” Our DLQ is a plain durable table: event ID, destination, payload, attempt history, and last error. Nothing fancy. What made it useful was building the replay tool at the same time as the queue, not months later. An event sitting in a DLQ that nobody can inspect or resubmit is just a slower way to lose data. We wired ours to page on-call once an endpoint accumulates more than 20 dead-lettered events in an hour, because that’s a much stronger signal of “this partner’s endpoint is actually down” than any single failed request. 问题的关键不在于“我们重试次数不够”，而在于“当重试耗尽时，事件就消失了”。我们的死信队列是一个简单的持久化表：包含事件 ID、目标地址、负载、尝试历史和最后一次错误。没什么花哨的。让它真正有用的是我们在构建队列的同时就开发了重放工具，而不是几个月后再做。如果死信队列里的事件没人能查看或重新提交，那只是另一种更慢的数据丢失方式。我们将系统设置为：一旦某个端点在一小时内积累了超过 20 个死信事件，就向值班人员发送告警，因为相比于单个请求失败，这更能明确地表明“该合作伙伴的端点确实挂了”。

Idempotency keys had to come first, not after

幂等键必须先行，而非事后补救

Retries only work if replaying a webhook is safe. We’d been lucky that most of our event handlers were naturally idempotent, but “most” isn’t a guarantee, and a payment-status webhook is exactly the kind of event where a duplicate delivery causing a duplicate downstream action is worse than the original failure. Every event we send now carries a UUID in the payload and header, and we document — loudly, in the integration guide — that consumers are expected to dedupe on it. That’s a contract change, not just an internal one, and we spent real time making sure existing partners knew before we shipped it. 只有在重放 Webhook 是安全的前提下，重试才有意义。我们很幸运，大多数事件处理器天生就是幂等的，但“大多数”并不能作为保证。支付状态 Webhook 正是那种重复发送导致下游重复操作比原始故障更糟糕的事件。现在我们发送的每个事件都在负载和头部携带一个 UUID，并且我们在集成指南中明确强调：消费者必须基于此进行去重。这是一个契约变更，而不仅仅是内部调整，我们在发布之前花了很多时间确保现有合作伙伴知晓这一点。

Distinguishing “retry this” from “don’t bother” mattered more than we expected

区分“重试”与“放弃”比我们预想的更重要

A 500 from a downstream service is worth retrying. A 400 because the payload is malformed is not — retrying it six times over two hours just delays the DLQ entry and the alert that would have caught the real bug faster. We now classify failures at send time and only apply backoff to the response codes that are actually likely to resolve on their own. 下游服务的 500 错误值得重试。但因为负载格式错误导致的 400 错误则不然——在两小时内重试六次只会延迟进入死信队列的时间，也会延迟本应更快发现真正 Bug 的告警。我们现在会在发送时对失败进行分类，仅对那些确实可能自行恢复的响应代码应用退避策略。

None of this is novel — retry-with-backoff-and-DLQ is a well-worn pattern. What’s easy to underestimate is how much of the value is in the boring parts: the replay tooling, the alerting thresholds, the idempotency contract with partners. The queue itself is a day of work. Making failures visible and recoverable instead of silent is the part that actually prevents the next 2 a.m. email from a customer telling you something broke hours ago. 这些都不是什么新鲜事——“带退避的重试 + 死信队列”是一个非常成熟的模式。容易被低估的是那些枯燥部分所蕴含的价值：重放工具、告警阈值以及与合作伙伴的幂等契约。构建队列本身只需一天的工作量。让故障变得可见且可恢复，而不是静默处理，这才是真正能防止你在凌晨两点收到客户邮件抱怨“几小时前就坏了”的关键所在。

We ended up rebuilding a good chunk of our event-delivery pipeline around these lessons at Edilec, and it’s now the default pattern we reach for anytime a service needs to guarantee delivery to something outside our control — you can see more of how we approach this kind of reliability work at edilec.com. 在 Edilec，我们最终围绕这些经验重构了大部分事件交付流水线。现在，每当服务需要向我们无法控制的系统保证交付时，这已成为我们的默认模式——你可以在 edilec.com 查看我们如何处理此类可靠性工作的更多信息。