Just Use Postgres for Durable Workflows

直接使用 Postgres 构建持久化工作流

Durable workflows are a simple but powerful tool for building reliable programs. The idea is that as your program runs, you regularly checkpoint its progress to a database. That way, if your program ever crashes or fails, you can reload from the last checkpoint to recover it from its last completed step. You can think of this like saving in a video game: you regularly “save” your program’s progress so that if it crashes, you can “reload” it from its last checkpoint.

持久化工作流（Durable workflows）是构建可靠程序的一种简单而强大的工具。其核心思想是：当程序运行时，定期将进度检查点（checkpoint）保存到数据库中。这样，如果程序崩溃或失败，你可以从上一个检查点重新加载，从而从最后完成的步骤恢复。你可以将其想象成电子游戏中的存档：定期“保存”程序的进度，以便在崩溃时能从上一个检查点“读取”进度。

Most commonly, durable workflows are implemented via external orchestration. This is the pattern used by systems like Temporal, Airflow, and AWS Step Functions. In this model, durable programs are written as workflows of steps whose execution is coordinated by a central orchestrator.

通常，持久化工作流是通过外部编排（external orchestration）来实现的。Temporal、Airflow 和 AWS Step Functions 等系统都采用了这种模式。在这种模型中，持久化程序被编写为一系列步骤的工作流，其执行由中央编排器进行协调。

When a client submits a workflow, the orchestrator creates a record for it in a data store then dispatches it to a worker for execution. Each time a worker completes a step, it sends the step’s outcome back to the orchestrator. The orchestrator checkpoints the output in its data store, then dispatches the next step. If a worker crashes or fails, the orchestrator dispatches its workflows to another worker, starting them from their last checkpointed step.

当客户端提交工作流时，编排器会在数据存储中为其创建记录，然后将其分发给工作节点（worker）执行。每当工作节点完成一个步骤，它会将结果发送回编排器。编排器将输出结果存入数据存储作为检查点，然后分发下一个步骤。如果工作节点崩溃或失败，编排器会将工作流重新分发给另一个节点，并从上一个检查点开始执行。

In this blog post, we’ll argue that external orchestration is fundamentally overcomplicated. The core idea of durable workflows is to checkpoint program state in a database. But if durable workflows are about databases, then there’s no reason to have a separate orchestrator server. Instead, it’s simpler and more efficient to use the database itself as an orchestrator. To make this more concrete, we’ll focus specifically on building durable workflows on Postgres, because its popularity, scalability, and rich ecosystem make it an ideal choice.

在这篇博文中，我们将论证外部编排本质上过于复杂。持久化工作流的核心思想是将程序状态检查点存入数据库。既然持久化工作流本质上是关于数据库的，那么就没有理由再额外维护一个编排器服务器。相反，直接使用数据库本身作为编排器会更简单、更高效。为了更具体地说明这一点，我们将重点讨论如何在 Postgres 上构建持久化工作流，因为其流行度、可扩展性和丰富的生态系统使其成为理想的选择。

In a Postgres-backed durable workflows system, application servers directly communicate with Postgres to execute workflows instead of going through a central orchestrator. A client submits a workflow for execution by creating an entry for it in a Postgres workflows table. Application servers poll the table for workflows to dequeue and execute. As a server executes a workflow, it checkpoints the output of each step to Postgres. If a server executing workflows crashes or fails, another server can recover its workflows from their checkpoints.

在基于 Postgres 的持久化工作流系统中，应用服务器直接与 Postgres 通信来执行工作流，而无需经过中央编排器。客户端通过在 Postgres 的工作流表中创建条目来提交工作流。应用服务器轮询该表以获取并执行工作流。当服务器执行工作流时，它会将每个步骤的输出检查点存入 Postgres。如果执行工作流的服务器崩溃或失败，另一台服务器可以从检查点恢复这些工作流。

This design renders a central orchestrator unnecessary because application servers can coordinate through Postgres. Instead of relying on a central orchestrator to dispatch workflows to workers, servers cooperatively dequeue workflows from a Postgres table, using mechanisms such as locking clauses to ensure each workflow is dequeued by exactly one worker. Instead of relying on an orchestrator to checkpoint step outputs, workers checkpoint steps to Postgres themselves. If multiple workers try to execute the same workflow simultaneously, Postgres database integrity constraints let them detect the duplicate work on checkpoint and back off.

这种设计使得中央编排器变得多余，因为应用服务器可以通过 Postgres 进行协调。服务器不再依赖中央编排器来分发工作流，而是协作地从 Postgres 表中取出工作流，并利用锁定子句（locking clauses）等机制确保每个工作流仅被一个工作节点取出。工作节点也不再依赖编排器来保存步骤输出，而是自行将检查点存入 Postgres。如果多个工作节点尝试同时执行同一个工作流，Postgres 的数据库完整性约束可以让他们在保存检查点时检测到重复工作并自动回退。

Replacing a central orchestrator with Postgres (or another database) makes durable workflows fundamentally simpler. In particular, it means hard problems such as scalability, availability, observability, and security can be addressed using well-understood Postgres-native solutions.

用 Postgres（或其他数据库）取代中央编排器，从根本上简化了持久化工作流。特别是，这意味着可扩展性、可用性、可观测性和安全性等难题，都可以使用成熟的 Postgres 原生解决方案来解决。

Scalability and Availability / 可扩展性与可用性

The scalability and availability of a database-backed durable workflows system are fundamentally determined by the underlying database. The system can scale horizontally by adding more worker servers, so its maximum capacity is determined by how quickly the database can process workflows. Similarly, workers are fungible and can freely recover each other’s state, so the system is available as long as the underlying database is available.

数据库驱动的持久化工作流系统的可扩展性和可用性，从根本上取决于底层数据库。系统可以通过增加更多工作服务器来实现水平扩展，因此其最大容量取决于数据库处理工作流的速度。同样，工作节点是可互换的，可以自由恢复彼此的状态，因此只要底层数据库可用，整个系统就是可用的。

When using Postgres specifically, this is beneficial because Postgres scalability and availability are well-studied problems with robust solutions. For scalability, a single Postgres server can vertically scale to handle tens of thousands of workflows per second, and further scaling can be achieved by using distributed (e.g., CockroachDB) or sharded Postgres. For availability, Postgres supports streaming replication with automatic failover and managed offerings provide multi-AZ deployments with high-availability SLAs out of the box. As a result, the decades of engineering work and research that have gone into operating Postgres at scale can translate directly to operating durable workflows.

特别是在使用 Postgres 时，其优势在于 Postgres 的可扩展性和可用性是经过充分研究且拥有成熟解决方案的问题。在可扩展性方面，单个 Postgres 服务器可以通过垂直扩展处理每秒数万个工作流，进一步扩展则可以使用分布式（如 CockroachDB）或分片式 Postgres。在可用性方面，Postgres 支持带有自动故障转移的流复制，且托管服务开箱即用，提供多可用区部署和高可用性 SLA。因此，数十年来在 Postgres 大规模运维方面的工程实践和研究成果，可以直接转化为持久化工作流的运维能力。

Observability / 可观测性

When using Postgres-backed durable execution, workflows and their steps are checkpointed to Postgres tables. This means observability is built-in: you can scan those checkpoints to monitor workflows in real time and visualize workflow execution. Postgres excels at this because virtually any workflow observability query can be expressed in SQL. For example, here’s a query to find all workflows that errored in the last month:

当使用基于 Postgres 的持久化执行时，工作流及其步骤会被存入 Postgres 表中。这意味着可观测性是内置的：你可以扫描这些检查点来实时监控工作流并可视化执行过程。Postgres 在这方面表现出色，因为几乎任何工作流的可观测性查询都可以用 SQL 表示。例如，以下是一个查找上个月所有出错工作流的查询：

A query like this might seem obvious, but it’s hard to overstate how powerful this is. It’s only possible because Postgres’s relational model lets you express complex filtering and analytical operations declaratively in SQL, leveraging decades of research in query optimization. Many systems with simpler data models, such as the key-value stores used by popular external orchestrators, have no such support. By storing workflow and step data in Postgres tables and augmenting them with secondary indexes for fast analytical queries, you get efficient observability from your durable execution “for free.”

这样的查询看起来很简单，但其强大之处难以言表。这之所以可行，是因为 Postgres 的关系模型允许你通过 SQL 声明式地表达复杂的过滤和分析操作，并利用了数十年的查询优化研究成果。许多数据模型更简单的系统（例如流行外部编排器所使用的键值存储）并不具备这种支持。通过将工作流和步骤数据存储在 Postgres 表中，并辅以二级索引以实现快速分析查询，你可以“免费”获得持久化执行的高效可观测性。

Reliability and Security / 可靠性与安全性

When using an external orchestrator for durable execution, both the orchestrator and its data store are single points of failure. Because they directly coordinate workflow execution, if either has downtime, the entire application becomes unavailable. Moreover, because they process and store workflow and step checkpoints, they likely have access to sensitive application data, meaning they must be hardened, access-controlled, and audited like any other piece of sensitive infrastructure. By contrast, the only point of failure in Postgres…

当使用外部编排器进行持久化执行时，编排器及其数据存储都是单点故障。由于它们直接协调工作流执行，如果其中任何一个停机，整个应用程序就会变得不可用。此外，由于它们处理并存储工作流和步骤的检查点，它们很可能能够访问敏感的应用程序数据，这意味着它们必须像其他任何敏感基础设施一样进行加固、访问控制和审计。相比之下，Postgres 中唯一的故障点是……