本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

One Hash Bloom Filters

Jim Calabro About RSS | Atom | JSON --- Bluesky GitHub Chess Last.fm One Hash Bloom Filters Oct 17, 2026 Note: as always, all my public writing is written by hand by me.

Jim Calabro 关于 RSS | Atom | JSON --- Bluesky GitHub Chess Last.fm 单哈希布隆过滤器 2026年10月17日 注：一如既往，我所有的公开文章均由我亲手撰写。

Overview A bloom filter is a probabilistic data structure that allows you to quickly and cheaply answer the question “is key X in this set?” with either “definitely not” or “probably”. You pick a false positive rate that is acceptable for your workload and hardware, add keys to the set, and check for their existence later on. Bloom filters use a small amount of memory compared to the cardinality of the set, and are computationally very cheap.

概述 布隆过滤器是一种概率数据结构，它能让你快速且低成本地回答“键 X 是否在这个集合中？”这一问题，答案要么是“绝对不在”，要么是“可能在”。你可以根据工作负载和硬件选择一个可接受的误报率，将键添加到集合中，并在之后检查它们是否存在。与集合的基数相比，布隆过滤器占用的内存很少，且计算成本非常低。

For these reasons, they’re frequently used in high performance data systems. For example, a database might keep an in-memory bloom filter and check it before actually going to disk when looking up rows. This saves substantial amounts of very slow disk IO. We also use them all over the place in the Bluesky backend. They allow us to save millions of queries per second against our Scylla database when looking up blocks, mutes, labels, and more. For instance, we can often answer the question “does user X block user Y” without even hitting the database.

由于这些原因，它们经常被用于高性能数据系统中。例如，数据库可能会保留一个内存中的布隆过滤器，并在查找行时先检查它，然后再真正访问磁盘。这节省了大量极其缓慢的磁盘 I/O。我们在 Bluesky 后端也到处使用它们。在查找屏蔽、静音、标签等信息时，它们使我们能够每秒减少对 Scylla 数据库的数百万次查询。例如，我们通常无需访问数据库就能回答“用户 X 是否屏蔽了用户 Y”的问题。

Here’s the last several days of some of our bloom filter positive hit rates: The above yellow line has a roughly 1% positive rate, which means that 99% of the time, we can completely skip querying our database. This has been critical in our ability to scale the site and operate the site cheaply.

以下是我们过去几天部分布隆过滤器的命中率情况：上方的黄线显示命中率约为 1%，这意味着在 99% 的情况下，我们可以完全跳过数据库查询。这对于我们扩展网站规模并以低成本运营网站至关重要。

Traditional Bloom Filters A bloom filter is an array of bits and k hash functions. You choose the size of the array and k before hand, and want to right-size it to your workload to achieve a specific false positive rate. To add a key to the set, run it through the k hash functions and set the bits at each of those locations. To check if a a key is in the set, hash it k times and check each resulting location in the bit array. If any bits are zero, the key was never added. If all bits are set to one, it might be in the set. Here’s an example of a potential positive: And a definite negative: Pretty simple!

传统布隆过滤器 布隆过滤器是一个位数组和 k 个哈希函数的组合。你需要预先选择数组的大小和 k 值，并根据工作负载调整其大小，以达到特定的误报率。要将键添加到集合中，需通过 k 个哈希函数运行它，并将每个位置的位设置为 1。要检查键是否在集合中，需对其进行 k 次哈希运算，并检查位数组中对应的每个位置。如果任何位为 0，则该键从未被添加过。如果所有位都为 1，则它可能在集合中。这是一个潜在命中的示例：以及一个绝对未命中的示例：非常简单！

Double Hashing While much cheaper than going to disk, computing k independent hashes per key is still a bit expensive for very high performance systems. In 2008, Kirsch and Mitzenmacher wrote a paper demonstrating that two hashes can get you the same output. The math is far over my head, but if you derive the i-th location as (hash_1 + i*hash_2) % n, then the false positive rate is effectively the same as using k independent hashes. Phew!

双重哈希 虽然比访问磁盘便宜得多，但对于超高性能系统来说，每个键计算 k 个独立哈希仍然有些昂贵。2008 年，Kirsch 和 Mitzenmacher 发表了一篇论文，证明了两个哈希值就能达到相同的输出效果。数学原理超出了我的理解范围，但如果你将第 i 个位置推导为 (hash_1 + i*hash_2) % n，那么误报率实际上与使用 k 个独立哈希是一样的。呼！

bits-and-blooms/bloom maintained by Daniel Lemire (a huge inspiration of mine, you should buy his book) is a fantastic double hash bloom filter implementation written in go. It runs 128-bit murmur3 twice to get four 64-bit values, then combines them to produce all k bit locations. We used this in production at Bluesky for a long time.

由 Daniel Lemire（我深受其启发，你应该买他的书）维护的 bits-and-blooms/bloom 是一个用 Go 语言编写的非常棒的双重哈希布隆过滤器实现。它运行两次 128 位 murmur3 以获得四个 64 位值，然后将它们组合以生成所有 k 个位位置。我们在 Bluesky 的生产环境中长期使用它。

For the Bluesky backend, we essentially do a SELECT * FROM blocks, mutes, list items, etc. every time one the processes starts, and add all the items to the bloom filter. There’s also a real time atproto firehose consumer inserting items in to bloom filters (even during process startup), so there is a race condition if you’re not careful. The problem with the bits-and-blooms library is that it’s not internally thread-safe, so we needed to wrap all bloom filter access in a mutex lest we suffer a data race.

对于 Bluesky 后端，我们基本上在每个进程启动时都会执行 SELECT * FROM blocks, mutes, list items 等操作，并将所有项目添加到布隆过滤器中。此外，还有一个实时的 atproto firehose 消费者在向布隆过滤器插入项目（即使在进程启动期间也是如此），因此如果不小心，就会出现竞态条件。bits-and-blooms 库的问题在于它内部不是线程安全的，所以我们需要用互斥锁包装所有布隆过滤器的访问，以免发生数据竞争。

pprof showed that mutex contention was actually the bottleneck in launching the process. A single deploy used to take 90 minutes to roll out fully because of this! So my colleague Jasmine forked the library and created atomic-bloom, which uses atomic CPU instructions to do the same logic, but safely and without mutexes. This led to substantially less contention and was massively faster than the mutex-based approach. Our deploys dropped from 90 minutes to about 15, which enabled us to ship faster and with greater confidence. Awesome!

pprof 显示，互斥锁争用实际上是启动进程时的瓶颈。正因如此，单次部署曾经需要 90 分钟才能完全上线！所以我的同事 Jasmine 分叉了该库并创建了 atomic-bloom，它使用原子 CPU 指令来实现相同的逻辑，但更安全且无需互斥锁。这大大减少了争用，并且比基于互斥锁的方法快得多。我们的部署时间从 90 分钟缩短到了约 15 分钟，这使我们能够更快、更自信地发布产品。太棒了！

One Hash Can we go further? For Bluesky, the answer is no. Contention was our original problem, not the number of hash functions or other factors, thus by Amdahl’s Law we did not receive any further speedup resulting in a noticeable impact. However, it’s still fun to go deeper and learn something new! As part of our research on bloom filter techniques, we came across this paper from Lu et al. that uses even more math to show how to do bloom filters using only a single hash.

单哈希 我们还能更进一步吗？对于 Bluesky 来说，答案是否定的。争用是我们最初的问题，而不是哈希函数的数量或其他因素，因此根据阿姆达尔定律，我们没有获得任何能产生显著影响的进一步加速。然而，深入研究并学习新知识仍然很有趣！作为我们对布隆过滤器技术研究的一部分，我们发现了 Lu 等人的这篇论文，它使用了更多的数学方法来展示如何仅使用单个哈希来实现布隆过滤器。

Here’s an overview of how it works: Instead of one big bit array, split that into k array partitions of fixed sizes Each partition has a different size, and all the sizes are pairwise coprime, meaning that no two sizes share a common factor other than one You hash the given key once to get hash The bit position in partition p is hash%size_p (the size of the partition) Since you have k partitions, you get k bit positions to check out of that single hash You can use those k bit positions as you would with other bloom filter types!

以下是其工作原理概述：不再使用一个大的位数组，而是将其拆分为 k 个固定大小的数组分区。每个分区的大小不同，且所有大小都是两两互质的，这意味着除了 1 之外，没有两个大小共享公因数。你对给定的键进行一次哈希运算以获得 hash。分区 p 中的位位置是 hash%size_p（该分区的大小）。由于你有 k 个分区，你可以从那单个哈希中获得 k 个要检查的位位置。你可以像使用其他类型的布隆过滤器一样使用这 k 个位位置！

With this technique in hand, I made jcalabro/gloom, a one-hash bloom filter (OHBF) implementation in go. This has been live in the Bluesky backend for the past 9 months and has been holding up quite well. It’s also used in jetstream, our high performance sync service. Here’s an example of how it’s used:

掌握了这项技术后，我制作了 jcalabro/gloom，这是一个用 Go 语言实现的单哈希布隆过滤器 (OHBF)。它已经在 Bluesky 后端运行了 9 个月，表现相当不错。它也被用于我们的高性能同步服务 jetstream 中。以下是其使用示例：

[Code block omitted]

[代码块省略]

Now, let’s step through how it works under the hood in more detail. Here’s some sample go code that shows how we select the block index as well as h for a given key we want to check: hash := xxh3.Hash128(key) // calculate the 1

现在，让我们更详细地了解其底层工作原理。以下是一些示例 Go 代码，展示了我们如何为想要检查的给定键选择块索引以及 h：hash := xxh3.Hash128(key) // 计算 1