Exactly-Once Semantics in Kafka: Promise vs. Reality

查看原文 / View Original 编辑 / Edit

Exactly-Once Semantics in Kafka: Promise vs. Reality

Kafka 的精确一次语义：承诺与现实

“We’re using Kafka with exactly-once semantics, so we don’t have to worry about duplicates.” I’ve heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka’s exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents.

“我们正在使用具有精确一次（Exactly-Once）语义的 Kafka，所以不必担心重复数据。” 我在架构评审、设计文档和故障复盘中经常听到这句话。这代表了对 Kafka 精确一次保证范围的误解，而这种承诺与现实之间的差距已经导致了真实的生产事故。

What Kafka’s Exactly-Once Actually Covers

Kafka 的精确一次语义究竟涵盖了什么

Kafka’s exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels: Kafka 在 0.11.0 版本中引入的精确一次语义（EOS）主要在两个层面运作：

  • Producer idempotence (enable.idempotence=true): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count.

  • 生产者幂等性 (enable.idempotence=true)： 生产者为每条消息分配一个序列号。Broker 会对具有相同生产者 ID 和序列号的消息进行去重。这防止了因生产者重试导致的重复——无论重试多少次，消息最终只会进入 Kafka 分区一次。

  • Transactions (transactional.id): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions. Together, these give you exactly-once message delivery within the Kafka cluster.

  • 事务 (transactional.id)： 允许生产者原子性地写入多个分区。要么所有写入都提交，要么都不提交。结合消费者的 isolation.level=read_committed 设置，读取者只能看到已提交的事务。两者结合，实现了 Kafka 集群内部的精确一次消息投递。

What Exactly-Once Does Not Cover

精确一次语义不涵盖什么

Here’s the boundary that engineers miss: Kafka’s exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you’re outside the transaction boundary. 这是工程师们容易忽略的边界：Kafka 的精确一次保证仅限于 Kafka 集群内部。一旦你的消费者在 Kafka 之外执行任何操作——比如写入数据库、调用 REST API 或发布到云队列——你就已经超出了事务边界。

Consider a typical consumer: 考虑一个典型的消费者：

consumer.poll(records);
for (record : records) {
    database.save(process(record)); // External write — outside Kafka transaction
}
consumer.commitSync();

If the application crashes after database.save() but before commitSync(), Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event. Enabling producer idempotence on the consumer’s Kafka writes does not fix this. 如果应用程序在 database.save() 之后但在 commitSync() 之前崩溃，Kafka 会重新投递该消息。消费者会重新处理它，导致数据库中出现两条针对同一事件的写入记录。在消费者的 Kafka 写入端启用生产者幂等性并不能解决这个问题。

The Patterns That Actually Give You End-to-End Safety

真正实现端到端安全性的模式

Idempotent Consumers 幂等消费者

Design consumer processing logic to be idempotent: processing the same message twice produces the same result as once. 设计幂等的消费者处理逻辑：处理同一条消息两次产生的结果与处理一次相同。

public void processPaymentEvent(PaymentEvent event) {
    if (paymentRepository.existsByEventId(event.getId())) {
        return; // Already processed
    }
    paymentRepository.save(toPayment(event));
}

Use a unique constraint on event_id and handle the constraint violation rather than relying solely on the pre-check — this closes the concurrent processing race condition. 在 event_id 上使用唯一约束，并处理约束冲突，而不是仅仅依赖预检查——这可以消除并发处理中的竞态条件。

Transactional Outbox 事务性发件箱 (Transactional Outbox)

Write to your database and record the outgoing Kafka message in an outbox table in the same local transaction: 在同一个本地事务中，将数据写入数据库并将待发送的 Kafka 消息记录在发件箱表中：

BEGIN;
INSERT INTO payments (id, amount, status, event_id) VALUES (...);
INSERT INTO outbox (event_id, topic, payload) VALUES (...);
COMMIT;

A separate process (Debezium CDC or a polling publisher) reads the outbox and publishes to Kafka. This makes the database write and Kafka publish effectively atomic from a business perspective. 通过一个独立的进程（如 Debezium CDC 或轮询发布器）读取发件箱并发布到 Kafka。从业务角度来看，这使得数据库写入和 Kafka 发布实际上是原子性的。

Offset-Aware Idempotency 感知偏移量的幂等性

Store the consumed offset alongside the processed result in the same database transaction. On restart, start consuming from the last successfully committed offset. This requires managing offsets manually rather than using auto-commit. 在同一个数据库事务中，将已消费的偏移量与处理结果一起存储。重启时，从最后一次成功提交的偏移量开始消费。这需要手动管理偏移量，而不是使用自动提交。

Performance Considerations

性能考量

Kafka transactions have a cost: min.insync.replicas=2 is required for EOS correctness — forcing synchronous replication, which increases write latency. Transaction coordinators add a round-trip per transaction commit. In payment event pipelines where I’ve implemented EOS, throughput dropped roughly 15-20% compared to at-least-once, with higher p99 latencies. In the context of financial correctness, that was an acceptable tradeoff — but it’s a tradeoff you should measure, not assume. Kafka 事务是有代价的：为了保证 EOS 的正确性，必须设置 min.insync.replicas=2，这会强制进行同步复制，从而增加写入延迟。事务协调器在每次事务提交时都会增加一次往返开销。在我实施 EOS 的支付事件流水线中，吞吐量相比“至少一次”（at-least-once）模式下降了约 15-20%，且 p99 延迟更高。在金融准确性的语境下，这是一个可以接受的权衡——但这是一个你应该去测量而非假设的权衡。

The Honest Summary

诚实的总结

Kafka’s exactly-once semantics is real and valuable — for the Kafka-to-Kafka case. Read from a topic, process, write results to another topic: that chain can be exactly-once if you configure it correctly. Kafka 的精确一次语义是真实且有价值的——针对 Kafka 到 Kafka 的场景。从一个主题读取、处理、并将结果写入另一个主题：如果你配置正确，这个链路可以实现精确一次。

For consumers that write to databases or call external services, exactly-once at the Kafka layer removes the easy duplicate vector (broker-level retries) but doesn’t remove the hard one (consumer restarts after partial processing). Your consumer still needs to be idempotent. 对于写入数据库或调用外部服务的消费者，Kafka 层的精确一次消除了简单的重复向量（Broker 级别的重试），但没有消除困难的向量（部分处理后消费者重启）。你的消费者仍然需要保持幂等性。

Design idempotent consumers. Use the transactional outbox for end-to-end atomicity. Use Kafka EOS as one layer in a defense-in-depth approach, not as a magic bullet. 设计幂等消费者。使用事务性发件箱来实现端到端的原子性。将 Kafka EOS 作为纵深防御体系中的一层，而不是将其视为万能药。

← 返回新闻列表 / Back to news list