@Cron fires on every replica. I found out the boring way

@Cron 在每个副本上都会触发，我以一种枯燥的方式发现了这一点

How an in-process scheduler turned one nightly job into N duplicate writes, and why I moved scheduling out of the app entirely. I was looking at staging data when I found two of something that should only ever exist once: two identical records for the same entity, same start date, created seconds apart. Not corrupted, not half-written. Just two, where there should have been one. The job that creates those records runs once a night. So why did it run twice? Because we run more than one copy of the service, and the scheduler lived inside the service.

一个进程内调度器是如何将一个每晚执行的任务变成 N 次重复写入的，以及我为什么将调度逻辑完全移出了应用程序。我在查看暂存环境（staging）数据时，发现了一些本应只存在一次的东西出现了两次：针对同一个实体、同一个开始日期，有两条完全相同的记录，创建时间仅相差几秒。它们没有损坏，也没有写入一半。本该只有一条，却出现了两条。创建这些记录的任务每晚只运行一次。那么为什么它运行了两次？因为我们运行了不止一个服务副本，而调度器就驻留在服务内部。

The setup

设置

A nightly job rolls every active entity forward into its next bounded time-window — one new record each, once a day. Standard NestJS:

一个每晚执行的任务会将每个活跃实体推进到下一个有界时间窗口——每天一次，每个实体生成一条新记录。标准的 NestJS 写法如下：

@Injectable()
export class WindowGenerationService {
  @Cron('0 8 * * *') // every day at 08:00
  async generateNextWindows() {
    const entities = await this.repo.findActiveEndingSoon();
    for (const entity of entities) {
      await this.repo.createNextWindow(entity);
    }
  }
}

This is the documented way to schedule work in NestJS. It’s what you reach for. It worked perfectly the entire time we ran a single instance. Then we scaled horizontally. Now there are three instances. @Cron doesn’t know that. The decorator schedules the job in every process that boots it. Three instances, three timers, three simultaneous runs at 08:00 — and because there was no idempotency check and no lock, all three sailed past “does this already exist?” and each wrote its own copy.

这是 NestJS 文档中推荐的调度任务方式，也是你最先想到的方法。在我们只运行单个实例时，它工作得非常完美。后来我们进行了水平扩展，现在有了三个实例。@Cron 并不知道这一点。装饰器会在启动它的每个进程中调度该任务。三个实例、三个计时器、在 08:00 同时运行三次——由于没有幂等性检查，也没有锁，所有三个实例都顺利通过了“这是否已经存在？”的检查，并各自写入了自己的副本。

Two things bothered me equally. One, duplicate data. Two, I was paying three containers to do the same work at the same time and then paying again to clean up the mess. The scheduler being in-process wasn’t just a correctness bug, it was wasted compute by design.

有两件事同样让我困扰。第一，重复数据。第二，我付钱让三个容器在同一时间做同样的工作，然后还要再付钱去清理烂摊子。调度器驻留在进程内不仅是一个正确性 Bug，而且在设计上就是对计算资源的浪费。

What I considered

我的考量

Option 1: a distributed lock. Keep @Cron, but have instances race for a lock — an advisory lock, or a key in a cache — and let the winner run. It works, and it trades a loud failure for a silent one. Duplicate writes are at least visible; you can find them and delete them. A missed run isn’t: if the lock backend is down at 08:00, or an instance dies holding the lock and the TTL hasn’t expired, the job just doesn’t fire, and nobody notices until someone’s window is missing days later. I’d be adding a whole new dependency, and a new way to fail quietly, to fix a problem I’d created by putting the timer in the wrong place.

方案 1：分布式锁。 保留 @Cron，但让实例竞争获取锁（咨询锁或缓存中的键），让获胜者运行。这确实有效，但它将一种明显的失败换成了一种静默的失败。重复写入至少是可见的；你可以找到并删除它们。而任务漏跑则不然：如果锁后端在 08:00 宕机，或者某个持有锁的实例在 TTL 过期前死亡，任务就不会触发，直到几天后有人发现窗口缺失时，才会被人察觉。为了修复我因将计时器放错位置而导致的问题，我却引入了一个全新的依赖项和一种新的静默失败方式。

Option 2: idempotency only. Add a check so a second run is a no-op. Cheap, correct, but it still wakes up three containers to do redundant work every night.

方案 2：仅实现幂等性。 添加检查，使第二次运行成为空操作（no-op）。成本低、正确，但它仍然会唤醒三个容器，每晚做重复的工作。

Option 3: take scheduling out of the app. The app shouldn’t own when. Let an external scheduler own the clock and fire a single HTTP request at a normal endpoint. The app only owns what happens when it’s poked. One trigger, one run, and horizontal scaling stops multiplying anything. I went with 3, and kept the idempotency from 2 as a seatbelt.

方案 3：将调度移出应用程序。 应用程序不应该负责“何时”执行。让外部调度器掌握时钟，并向一个普通端点发送单个 HTTP 请求。应用程序只负责在被触发时执行什么。一次触发，一次运行，水平扩展就不会再导致任何东西被成倍增加。我选择了方案 3，并保留了方案 2 中的幂等性作为安全带。

The change

变更

The @Cron decorator went away. The logic moved behind an endpoint:

@Cron 装饰器被移除了。逻辑被移到了一个端点之后：

@Post('jobs/run')
async runJob(@Body() body: RunJobDto) {
  this.assertValidSecret(body.secret);
  return this.jobs.run(body.jobKey);
}

An external scheduler fires this once at 08:00. The instance that receives it runs the job; the other two never hear about it. The idempotency check stayed, because “fires once” is a promise infrastructure makes and occasionally breaks — retries, manual re-triggers, a restart mid-run:

外部调度器在 08:00 触发一次。接收到请求的实例运行任务；另外两个实例根本不知道这件事。幂等性检查被保留了下来，因为“只触发一次”是基础设施做出的承诺，但偶尔会失效——比如重试、手动重新触发、运行中途重启等情况：

async createNextWindow(entity: Entity) {
  const existing = await this.repo.findByEntityIdAndStartDate(
    entity.id,
    entity.nextStartDate,
  );
  if (existing) return; // already done, no-op
  await this.repo.create(/* ... */);
}

What it cost

代价

The moment you turn a private nightly job into an HTTP endpoint, you’ve created a public button that runs a job. Anyone who finds it can hammer it. So the endpoint is guarded by a shared secret — and the comparison matters more than people expect:

当你把一个私有的每晚任务变成一个 HTTP 端点时，你就创建了一个可以运行任务的公共按钮。任何发现它的人都可以对其进行攻击。因此，该端点由一个共享密钥保护——而且比较方式比人们预期的更重要：

private assertValidSecret(provided: string) {
  const expected = this.config.cronSecret;
  const a = Buffer.from(provided);
  const b = Buffer.from(expected);
  if (a.length !== b.length || !timingSafeEqual(a, b)) {
    throw new UnauthorizedException();
  }
}

A naive provided === expected leaks information through how long it takes to fail — a timing oracle you can walk character by character. timingSafeEqual compares in constant time. It’s a small thing that reviewers usually don’t flag; I’d rather not gamble that the endpoint stays undiscovered.

天真的 provided === expected 会通过失败所需的时间泄露信息——这是一种可以逐字符破解的计时预言机（timing oracle）。timingSafeEqual 在恒定时间内进行比较。这是一个审查者通常不会注意的小细节；我宁愿不赌这个端点能一直不被发现。

The other cost was dumber: picking the hour. The job has to run after everyone’s start of the day, and “everyone” spans several US time zones — a time that’s mid-morning in the east is pre-dawn out west. So the schedule is pinned to a fixed UTC hour chosen against the westernmost operating timezone, because that’s the one that makes it safe everywhere. UTC in the crontab, human timezones in the requirement — the gap between those two is where the real decision lives.

另一个代价更愚蠢：选择时间。任务必须在每个人的工作日开始后运行，而“每个人”跨越了多个美国时区——东部时间的上午可能是西部的黎明前。因此，调度被固定在一个根据最西端运营时区选择的 UTC 小时上，因为这样在任何地方都是安全的。Crontab 中使用 UTC，需求中使用人类时区——这两者之间的差距才是真正需要决策的地方。

The lesson

教训

An in-process scheduler is a lie the moment you run more than one instance — it quietly multiplies by your replica count, and nothing in the code hints at it. Scheduling is a “when” concern, and it belongs outside the app; the app should only own “what happens.” And the instant you expose a job as an endpoint to get there, it’s a public endpoint — treat it like one, and keep the idempotency check anyway, because “it only fires once” is someone else’s promise.

一旦你运行了不止一个实例，进程内调度器就是一个谎言——它会悄悄地按你的副本数量成倍增加，而代码中没有任何迹象表明这一点。调度是一个关于“何时”的问题，它属于应用程序之外；应用程序应该只负责“发生了什么”。而当你为了实现调度将任务暴露为端点的那一刻起，它就是一个公共端点——请像对待公共端点一样对待它，并保留幂等性检查，因为“它只触发一次”是别人的承诺。