Fintech Shipment Fan-Out: SaaS Retention Cleanup and the Node.js Cron-Queue Boundary

金融科技货运分发：SaaS 保留数据清理与 Node.js 定时任务/队列边界

Short answer: use a scheduled cleanup endpoint when one indexed, bounded pass can finish predictably; use a queue when cleanup must be divided into independently retriable batches. For a fintech SaaS that fans out shipment updates to many subscribers, latency and cost should be judged at the system boundary: a cheap cleanup run is not a good bargain if it contends with delivery or leaves retention evidence incomplete. 简短的回答是：当一次有索引、有边界的清理任务可以预见地完成时，使用定时清理接口；当清理任务必须拆分为可独立重试的批次时，使用队列。对于向多个订阅者分发货运更新的金融科技 SaaS 而言，延迟和成本应在系统边界处进行评估：如果一次廉价的清理运行与交付任务产生资源竞争，或者导致保留数据证据不完整，那么它就不是一个划算的交易。

The first design decision is to keep shipment fan-out separate from retention work. A shipment update has a latency-sensitive path. Expired subscriptions, old delivery attempts, and temporary fan-out records usually have a policy-driven path. They may share a database, but they should not share an unbounded transaction or an execution budget. This distinction matters more than the spelling of a cron expression. It also gives the team a useful test: can the cleanup be repeated safely while the shipment update path continues to make progress? 首要的设计决策是将货运分发与数据保留清理工作分离开来。货运更新具有延迟敏感的路径。过期的订阅、旧的交付尝试以及临时的分发记录通常遵循策略驱动的路径。它们可能共享同一个数据库，但不应共享无边界的事务或执行预算。这种区分比 cron 表达式的写法重要得多。这也为团队提供了一个有用的测试：在货运更新路径持续进行的同时，清理工作能否安全地重复执行？

How should a Node.js SaaS choose a cron or queue for scheduled cleanup? Measure the worst case first. Count eligible records by tenant, check the relevant index, estimate lock pressure, and measure a bounded pass while the database is serving normal shipment traffic. The median duration is not the decision variable; the tail is. Node.js SaaS 应该如何为定时清理选择 cron 或队列？首先要测量最坏情况。按租户统计符合条件的记录，检查相关索引，评估锁压力，并在数据库处理正常货运流量的同时测量一次有边界的清理过程。中位数持续时间不是决策变量，长尾延迟才是。

A scheduled data cleanup is a good fit for one HTTP-triggered run when its cutoff, tenant scope, batch size, and completion state can be recorded and the run has room to finish before its execution limit. The cutoff should be computed by the application and persisted with the run. A schedule has jitter, and a paused schedule may not replay every missed invocation. “Delete records older than the cutoff captured at run start” is therefore more auditable than silently recalculating the boundary for every page. 当清理任务的截止日期、租户范围、批次大小和完成状态可以被记录，且任务有足够空间在执行限制内完成时，HTTP 触发的单次运行非常适合定时数据清理。截止日期应由应用程序计算并随运行记录持久化。调度存在抖动，且暂停的调度可能不会重放所有错过的调用。因此，“删除在运行开始时捕获的截止日期之前的记录”比为每一页静默重新计算边界更具可审计性。

The query should also exclude legal holds, active disputes, and any retention exception required by the business policy. Keep it bounded. The boundary is operational. When a tenant can monopolize a scan, when the pessimistic duration approaches the execution limit, or when one failed slice should not restart the entire pass, let the scheduled trigger produce work for queue consumers. Cron still supplies the clock; workers supply the execution boundary. 查询还应排除法律保留、活跃争议以及业务策略要求的任何保留例外。保持边界明确。边界是操作层面的。当某个租户可能独占扫描、悲观持续时间接近执行限制，或者某一个失败的分片不应导致整个过程重启时，应让定时触发器为队列消费者生成工作。Cron 依然提供时钟，而工作进程提供执行边界。

A process-local Node.js timer does neither reliably in a multi-replica deployment, because each replica can make its own decision about when to run. Consider a cleanup run with tenant-17 as the largest scope. The producer records one cutoff and emits a sequence of batch identities, each representing a bounded ordinal rather than a mutable list of row IDs. A worker claims the next slice using the eligibility predicate and a limit, records the claim, and commits the expiration transition with its completion marker. 在多副本部署中，进程内的 Node.js 定时器无法可靠地完成上述任务，因为每个副本都可以自行决定何时运行。考虑以 tenant-17 作为最大范围的清理运行。生产者记录一个截止日期并发出一个批次标识序列，每个标识代表一个有边界的序号，而不是可变的行 ID 列表。工作进程使用资格谓词和限制来认领下一个分片，记录认领状态，并提交带有完成标记的过期转换。

A retry does not need to know whether the first attempt reached the database, lost its connection after commit, or was made visible again after its timeout; it checks the marker and the conditional state, then reports a duplicate completion as a normal outcome. Meanwhile, a separate delivery worker can continue to process shipment updates because cleanup concurrency is capped and because the cleanup query uses the intended index rather than scanning the delivery history without a boundary. 重试机制不需要知道第一次尝试是否到达了数据库、是否在提交后丢失连接，或者是否在超时后重新可见；它只需检查标记和条件状态，然后将重复完成报告为正常结果。与此同时，独立的交付工作进程可以继续处理货运更新，因为清理并发受到限制，且清理查询使用的是预期的索引，而不是在没有边界的情况下扫描交付历史。

If the tenant has more eligible records than one run can handle, the next scheduled run continues from recorded state or creates the next immutable batch set, depending on the storage model. Neither choice should silently move the cutoff, because doing so makes the audit trail ambiguous: an operator could no longer tell whether a record was outside policy at the first run or merely missed by a later page. This is the kind of detail that makes a queue worthwhile, but it is also the detail that makes a queue expensive to operate. 如果租户拥有的符合条件的记录超过了一次运行的处理能力，下一次定时运行将根据存储模型从记录的状态继续，或创建下一个不可变的批次集。这两种选择都不应静默移动截止日期，因为这样做会使审计追踪变得模糊：操作员将无法判断一条记录是在第一次运行时就超出了策略范围，还是仅仅被后续页面遗漏了。这就是让队列变得有价值的细节，但也是让队列运维变得昂贵的细节。

Why do old records need idempotency and an audit trail? Deletion is an irreversible business effect, even if the database operation itself is ordinary. Exactly-once transport is not a sound assumption. The stronger and more practical invariant is an exactly-once business effect: a retry must find an already completed state and become a no-op, or it must apply the same conditional transition without changing the result. 为什么旧记录需要幂等性和审计追踪？删除是一种不可逆的业务影响，即使数据库操作本身很普通。精确一次（Exactly-once）传输并不是一个可靠的假设。更强大且更实用的不变性是“精确一次业务影响”：重试必须发现已完成的状态并变为无操作（no-op），或者必须在不改变结果的情况下应用相同的条件转换。

Give each cleanup run a stable identifier, cutoff, tenant scope, batch identity, attempt count, and outcome. Store discovered, processed, skipped, and remaining counts separately. Those numbers answer different reconciliation questions. A single success flag cannot show whether the producer stopped before batch 12, whether a worker received it twice, or whether a transaction committed the state marker but not the intended record transition. 为每次清理运行提供稳定的标识符、截止日期、租户范围、批次标识、尝试次数和结果。分别存储已发现、已处理、已跳过和剩余的计数。这些数字回答了不同的对账问题。单一的成功标志无法显示生产者是否在第 12 批之前停止、工作进程是否接收了两次，或者事务是否提交了状态标记但未提交预期的记录转换。

For a queue, publish a compact reference containing the tenant, cutoff, and batch ordinal rather than copying all records into the message. A worker can load the current eligible set, claim a bounded slice, and commit its completion marker with the state change where the storage model permits it. Redelivery then encounters the marker and exits cleanly. I’ve found that this evidence is more valuable than a promise that a particular scheduler will never retry. 对于队列，发布包含租户、截止日期和批次序号的紧凑引用，而不是将所有记录复制到消息中。工作进程可以加载当前符合条件的集合，认领一个有边界的分片，并在存储模型允许的情况下提交带有状态变更的完成标记。重新投递时会遇到该标记并干净地退出。我发现这些证据比“某个调度器永远不会重试”的承诺更有价值。

There is a second boundary when the cleanup signs callbacks or batch manifests. Use a keyed construction rather than treating a hash of public fields as authentication. RFC 2104 defines HMAC as a keyed-hash mechanism; the key belongs in secret management, and the signed material should include the immutable cutoff and batch identity. Authentication does not make deletion reversible, so it cannot replace legal holds or a carefully scoped eligibility query. 当清理任务签署回调或批次清单时，存在第二个边界。使用带密钥的构造，而不是将公共字段的哈希值视为身份验证。RFC 2104 将 HMAC 定义为带密钥的哈希机制；密钥应存放在密钥管理系统中，且签名材料应包含不可变的截止日期和批次标识。身份验证并不能使删除变得可逆，因此它不能替代法律保留或经过仔细限定的资格查询。

package main

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"strconv"
	"time"
)

func batchID(key []byte, tenant string, cutoff time.Time, ordinal int) string {
	mac := hmac.New(sha256.New, key)
	fmt.Fprintf(mac, "%s\x00%s\x00%s", tenant, cutoff.UTC().Format(time.RFC3339), strconv.Itoa(ordinal))
	return hex.EncodeToString(mac.Sum(nil))
}

func main() {
	cutoff := time.Date(2026, 8,