One Incident, Written Up Properly

一次事故，一次规范的复盘

Automatic top-up — the feature that charges a saved card when a customer’s balance falls below their threshold — could never have succeeded for anybody. The invoice was constructed in the wrong currency, and every attempt would have failed in a way that told the customer their card was bad. This is the whole write-up, in the shape we would want any incident written in.

自动充值功能——即当客户余额低于阈值时自动扣除已保存银行卡金额的功能——原本根本不可能成功。由于发票生成的币种错误，每一次尝试都会失败，且失败的提示会误导客户认为是他们的银行卡有问题。以下是完整的复盘报告，采用了我们期望中任何事故复盘应有的格式。

Summary

摘要

An invoice does not take its currency from the line items attached to it. It takes it from the customer’s default currency, or failing that from the Stripe account’s — which is EUR for a Dutch business. Every price in this product is denominated in USD. Finalising the invoice therefore failed with a currency-conflict error, on every automatic top-up, unconditionally. The manual top-up path was never affected, because a Checkout Session takes its currency from the first line item rather than from the customer record. That difference is why the bug could exist in a product whose payment flow demonstrably worked.

发票的币种并非取自其关联的明细项，而是取自客户的默认币种；如果客户没有设置，则取自 Stripe 账户的默认币种——对于一家荷兰企业来说，默认币种是欧元 (EUR)。而该产品的所有定价均以美元 (USD) 计算。因此，在自动充值时，发票结算会无条件地因币种冲突错误而失败。手动充值路径则不受影响，因为结账会话 (Checkout Session) 的币种取自第一个明细项，而非客户记录。正是这种差异，导致该 Bug 能够存在于一个支付流程看似完全正常的产品中。

Impact

影响

Dimension Description 维度描述

Customers affected: None. The defect was found before the path carried real traffic. This is stated plainly rather than omitted, because a postmortem that lets a near miss read as an outage is as dishonest as one that hides an outage.

受影响客户： 无。该缺陷在路径投入实际流量之前就被发现了。这一点必须明确说明，而不是略过，因为如果将“险些发生的事故”写成“服务中断”，其不诚实程度等同于隐瞒一次真正的事故。

What would have happened: Every automatic top-up fails. The failure surfaces as a payment error, which the failure counter records as a strike, and after three strikes the customer’s automatic top-up is switched off entirely.

如果发生会怎样： 所有的自动充值都会失败。失败表现为支付错误，失败计数器会记录一次违规，累计三次后，客户的自动充值功能将被彻底关闭。

What the customer would have concluded: That their card was declined. The message they receive says the saved card could not be charged. They would have gone and fixed a card that was working perfectly.

客户会得出什么结论： 他们的卡被拒付了。他们收到的消息显示无法扣除已保存的卡。他们会去“修复”一张原本完全正常的卡。

Secondary effect: A customer relying on automatic top-up to keep a production integration serving would have run out of credit silently, at whatever hour their traffic happened to cross the threshold. The second and third rows are what make this worth writing up. A defect that fails loudly and correctly is a bug; a defect that fails while blaming the user is a different category, because it consumes the customer’s time and trust on the way past.

次生影响： 依赖自动充值来维持生产环境集成的客户，会在其流量触及阈值的任何时刻，在毫无察觉的情况下耗尽余额。第二和第三点正是此次复盘的价值所在。一个能清晰且正确报错的缺陷只是一个 Bug；而一个通过责怪用户来报错的缺陷则属于另一个范畴，因为它在故障发生的过程中消耗了客户的时间和信任。

Timeline

时间线

The feature ships as part of a batch that also included profile pictures, two-factor authentication and email change. It is exercised by hand and by unit tests, both of which pass. A webhook-half harness is written. It drives a genuinely signed request through the real handler against a throwaway database copy, and asserts the things that lose money quietly: that a settled invoice credits exactly the amount bought, that a replay moves nothing, that two simultaneous deliveries of the same invoice move the balance once, that three declines switch the rule off. All pass.

该功能是作为包含个人资料图片、双重身份验证和电子邮件更改的一批功能的一部分发布的。它经过了手动测试和单元测试，两者均通过。我们编写了一个 Webhook 端的测试工具，它通过真实的处理器向一次性的数据库副本发送经过真实签名的请求，并验证那些会悄悄导致资金损失的逻辑：已结算的发票是否准确记入购买金额、重放请求是否不会产生变动、同一发票的两次同时交付是否只变动一次余额、三次拒付是否会关闭规则。所有测试均通过。

That harness names what it cannot prove, in the file itself: that Stripe accepts the invoice we build, that tax is applied to it, and that the mandate is accepted off-session. All three need a test-mode key and a card. A second check is written to close exactly that gap — the full path against real Stripe in test mode. Its first run fails immediately, at invoice finalisation, with a currency conflict. The defect had been present since the feature shipped. The interesting entry is the third. The gap was known, written down in the place a reader would find it, and closed. That is the mechanism that found this — not vigilance, and not luck.

该测试工具在文件本身中指出了它无法证明的内容：Stripe 是否接受我们构建的发票、税费是否已应用、以及授权是否在离线状态下被接受。这三者都需要测试模式的密钥和银行卡。我们编写了第二个检查程序来专门填补这一空白——即在测试模式下针对真实的 Stripe 环境进行全路径测试。它第一次运行就立即失败了，在发票结算阶段报出了币种冲突。该缺陷自功能发布以来就一直存在。有趣的是第三点：这个漏洞是已知的，并被记录在读者可以找到的地方，最终被修复了。这就是发现问题的机制——不是靠警惕性，也不是靠运气。

What actually happened

实际发生了什么

The direct cause is a mismatch between two currency defaults that never meet in any other code path: 直接原因是两个在其他代码路径中从未交集的默认币种设置不匹配：

Manual top-up (worked): Checkout Session -> line_items[0].price_data.currency = “usd” -> the SESSION currency is set from that line -> no conflict, ever.
手动充值（正常）： 结账会话 -> line_items[0].price_data.currency = “usd” -> 会话币种由该行设置 -> 永远不会冲突。
Automatic top-up (could never work): Invoice Item (currency: usd) + Invoice (currency: taken from customer.currency, else from the platform account default = eur) -> finalize -> “You cannot combine currencies on a single invoice … usd conflicts with the invoice currency eur”
自动充值（永远无法工作）： 发票明细项 (币种: usd) + 发票 (币种: 取自 customer.currency，否则取自平台账户默认值 = eur) -> 结算 -> “你不能在同一张发票上合并币种……usd 与发票币种 eur 冲突”。

There is no configuration in which this succeeds. It is not a race, not an edge case, not dependent on the amount. Two ways of creating a charge that look equivalent from the application’s side have different rules about where the currency comes from, and only one of them was ever exercised against the real service.

没有任何配置能让它成功。这不是竞态条件，不是边缘情况，也不取决于金额。从应用程序的角度看，两种创建扣款的方式看起来是等价的，但它们关于币种来源的规则却不同，而且只有其中一种在真实服务中经过了测试。

Why it was invisible

为什么它不可见

Four independent reasons, and they are the transferable part of this write-up. 四个独立的原因，这也是本次复盘中最具参考价值的部分。

The tested half was ours. The webhook harness covered the half of the path where our code decides things — crediting, deduplication, replay, the failure counter. That is genuinely the half that loses money quietly, and it was the right half to test first. It is also the half that cannot see a defect on the other side of the boundary. 测试的部分是我们自己的代码。 Webhook 测试工具覆盖了我们代码做决策的那一半路径——记账、去重、重放、失败计数器。这确实是容易悄悄导致资金损失的一半，也是最应该优先测试的一半。但这也是无法看到边界另一侧缺陷的一半。
The working sibling path created false confidence. Manual top-up worked, in production, repeatedly. Anyone reasoning about “can this product charge a card” had a demonstrably true answer to a slightly different question. 正常的同类路径制造了虚假的信心。 手动充值在生产环境中反复运行正常。任何推断“该产品能否扣除银行卡金额”的人，都会得到一个虽然正确但针对的是略有不同问题的答案。
The default came from account configuration, not code. No source file anywhere contains the string that caused this. The currency was inherited from a business registered in the Netherlands, which is not a fact any code review would surface. 默认值来自账户配置，而非代码。 没有任何源文件包含导致此问题的字符串。该币种是从一家在荷兰注册的企业继承而来的，任何代码审查都无法发现这一事实。
The failure had a plausible innocent explanation. An invalid-request error from a payment provider, mapped to “the saved card could not be charged”, is exactly what a genuinely declining card looks like. Had this reached production, the monitoring would have shown a card-failure rate, not a bug. The fourth is the one that generalises furthest. When an error path maps several distinct causes onto one customer-facing message, the least likely cause becomes indistinguishable from the most likely one, and the most likely one is what everybody investigates. 故障有一个看似合理的“无辜”解释。 来自支付提供商的无效请求错误，被映射为“无法扣除已保存的卡”，这看起来完全就像是一张真的被拒付的卡。如果这进入了生产环境，监控显示的将是“银行卡失败率”，而不是一个 Bug。第四点是最具普适性的：当错误路径将多个不同的原因映射到同一个面向客户的消息时，可能性最小的原因就会变得与可能性最大的原因无法区分，而每个人都会去调查那个可能性最大的原因。

What was changed

做了哪些修改

The currency is now set explicitly on the invoice rather than inherited, so the value is a property of the code rather than of an account setting somebody may change later. The new check drives the whole path against real Stripe in test mode. Stripe accepts the invoice, pays it off-sess… 现在，发票的币种被显式设置，而不是继承而来，因此该值成为了代码的一个属性，而不是以后可能会被某人更改的账户设置。新的检查程序在测试模式下针对真实的 Stripe 环境运行了整个路径。Stripe 接受了发票，并完成了离线支付……