Job queues are deceptively tricky

任务队列远比看起来复杂

One of the fun things about being a programmer is that as I look more into systems that I didn’t know much about, what superficially appears to be a simple system actually reveals interesting facets of underlying complexity. In other words, reality has a surprising amount of detail. 作为一名程序员，最有趣的事情之一就是当我深入研究那些我不太了解的系统时，那些表面上看起来简单的系统往往会展现出其底层复杂性中引人入胜的一面。换句话说，现实世界中蕴含着令人惊讶的细节。

In this post, I want to talk about job queues, which I’ve been thinking about for the past few days (and while drafting this post in my head, I realized I’d thought about them for longer at a previous job, but not nearly with as much clarity.) What do I mean by “job queue”? I mean a system where there is some notion of submitting batch jobs, scheduling them, and running them. Generally, the system is expected to be FIFO or FIFO-like, but that’s not required. Usually, the queue bundles together a native way to schedule jobs periodically – this lets you specify submissions using configuration files (e.g. using JSON or YAML). 在这篇文章中，我想谈谈任务队列（Job Queues）。过去几天我一直在思考这个问题（在脑海中构思这篇文章时，我意识到在上一份工作中我就思考过它们，但当时并没有现在这么清晰）。我所说的“任务队列”是指一个包含提交批处理任务、调度任务以及运行任务等概念的系统。通常，该系统被期望是先进先出（FIFO）或类似 FIFO 的，但这并非强制要求。通常，队列会内置一种定期调度任务的方法——这允许你使用配置文件（例如 JSON 或 YAML）来指定提交任务。

Job queues arise naturally in all sorts of situations where throughput requirements are high, but latency requirements are not that high. Continuous integration is a common example. Another example is summaries for the purpose of data analysis. 任务队列自然地出现在各种吞吐量要求高，但延迟要求不那么高的场景中。持续集成（CI）就是一个常见的例子。另一个例子是用于数据分析的汇总任务。

Over the past year or two, a handful of lenses that I’ve found useful when thinking about system design are: 在过去的一两年里，我在思考系统设计时发现以下几个视角非常有用：

Being wary of queues: Queues tend to typically be close to full or close to empty, requiring some thought around appropriate sizing. I’ve also learnt a bunch of counter-intuitive behavior of queues when it comes to latency from Marc Brooker’s blog. In blog posts I’ve read about queues, the focus is generally on latency, not on throughput, which is I think partly why I never really mentally linked “queues” and “job queues” earlier. 警惕队列： 队列通常要么接近满载，要么接近空闲，这需要对规模大小进行审慎思考。我还从 Marc Brooker 的博客中学到了许多关于队列在延迟方面的反直觉行为。在我读过的关于队列的博文中，重点通常是延迟而非吞吐量，我想这部分解释了为什么我之前从未将“队列”和“任务队列”在思维上联系起来。
Limits: This is based on the Tiger Style guide written by the folks at TigerBeetle, which talks about explicit limits on various things. On generalization, this leads to the notion of tolerances and having budgets for modular reasoning. Funnily enough, near the start of my career at Apple, I remember being mildly bemused at certain discussions where teams would discuss how much memory budget they would have to bargain for, especially for low-level code. I have grown more respect for that approach over time. 限制： 这基于 TigerBeetle 团队编写的《Tiger Style》指南，其中讨论了对各种事物设置明确的限制。推广开来，这引出了容差（tolerances）的概念，以及为模块化推理预留预算。有趣的是，在我职业生涯初期于苹果工作时，我记得当时对某些讨论感到有些困惑，团队会讨论他们需要争取多少内存预算，尤其是针对底层代码。随着时间的推移，我越来越尊重这种方法。
Fault models: Roughly speaking, a fault model describes your assumptions related to errors and reliability in your dependencies. I’ve learnt a bunch on this topic from the talks and tweets by the TigerBeetle CEO Joran Dirk Greef, as well as writing by Alex Miller. I will try to demonstrate how these lenses can apply for system design later in this post. 故障模型： 粗略地说，故障模型描述了你对依赖项中错误和可靠性的假设。我从 TigerBeetle 首席执行官 Joran Dirk Greef 的演讲和推文，以及 Alex Miller 的文章中学到了很多关于这个主题的知识。稍后我将尝试演示这些视角如何应用于系统设计。

The problem at hand

当前的问题

At $WORK, we have a background job for packing “reference repos”. A reference repo is a git repo which has been aggressively repacked – if you’re unfamiliar with repacking, you can think of this as more “aggressively compressed”. These are stored in object storage, and a new repo can be set up on a machine by downloading the reference repo, and fetching a delta of changes for the tip of the default branch. When it comes to very large repos, for the downloader, this approach helps cut down latency compared to the more typical approach of doing a git clone operation. 在工作中，我们有一个用于打包“参考仓库”（reference repos）的后台任务。参考仓库是一个经过激进重打包（aggressively repacked）的 Git 仓库——如果你不熟悉重打包，可以将其理解为更“激进的压缩”。这些仓库存储在对象存储中，通过下载参考仓库并获取默认分支顶端的变更增量，可以在机器上快速建立一个新的仓库。对于超大型仓库，这种方法相比传统的 git clone 操作，能显著降低下载器的延迟。

The actual details of how git does aggressive repacking is not super relevant for this post. What does matter is that there are, roughly speaking, two forms of repacking available: Git 如何进行激进重打包的具体细节与本文关系不大。重要的是，大致有两种可用的重打包形式：

Wholesale repacking: This will lead to git ignoring any baseline information about how packing should be done, and recompute everything from scratch. Say for the repo under test, this takes 7 hours. If you’re surprised at the thought of a git operation taking several hours, (1) you should be happy that you don’t have this problem and (2) some of the sub-operations here are seemingly single-threaded, regardless of repo size. 全量重打包： 这会导致 Git 忽略关于打包方式的任何基准信息，并从头开始重新计算所有内容。假设对于测试中的仓库，这需要 7 小时。如果你对一个 Git 操作需要几个小时感到惊讶，(1) 你应该庆幸自己没有遇到这个问题，(2) 这里的一些子操作似乎是单线程的，无论仓库大小如何。
Incremental repacking: This will lead to git reusing the baseline information already stored in the repo about packing, so it will only repack things which have changed. So if you did a clone, then did a fetch, then only the newer changes from fetch will be repacked, and only some cursory checks will be performed on the history received from the clone operation. Say this takes 2 hours. 增量重打包： 这会导致 Git 重用仓库中已存储的打包基准信息，因此它只会对发生变更的部分进行重打包。所以，如果你执行了 clone，然后执行了 fetch，那么只有 fetch 带来的新变更会被重打包，而对 clone 操作接收到的历史记录只会进行一些粗略的检查。假设这需要 2 小时。

OK. So that’s the setup. We have the option of doing the more expensive thing, which takes 7 hours. This buys us a smaller reference repo, which means faster downloads, faster un-tarring and lower disk usage. To give a size ballpark, a wholesale repacked repo can be up around 50-60% smaller than an incrementally packed repo. 好了，这就是背景。我们可以选择成本更高的方案，耗时 7 小时。这能为我们带来更小的参考仓库，意味着更快的下载速度、更快的解压速度以及更低的磁盘占用。粗略估计，全量重打包的仓库比增量重打包的仓库体积小约 50-60%。

We have the option of doing the cheaper thing, which takes 2 hours. This buys us more up-to-date reference repos, which means that if the downloader still cares about getting the latest changes, the delta it will fetch subsequently will be smaller, so it’ll be faster and put less load on the server. But also, if the extra disk usage is in the hundreds of megabytes or gigabytes, then that’s not great. One more thing to note is that since this is a code repository, downstream consumers are much more active on weekdays than on weekends. 我们也可以选择成本较低的方案，耗时 2 小时。这能为我们带来更新的参考仓库，意味着如果下载器需要获取最新变更，后续获取的增量会更小，从而速度更快，对服务器的负载也更小。但另一方面，如果额外的磁盘占用达到几百兆甚至几千兆字节，那就不太理想了。还需要注意的一点是，由于这是一个代码仓库，下游消费者在工作日的活跃度远高于周末。

The best of both worlds?

两全其美？

A natural next step upon seeing the above dichotomy is to propose the following: why not do the wholesale repacking on the weekends, and do the incremental repacking on the weekdays? That seems like it buys up-to-dateness on weekdays (where more consumers are active), and the wholesale repacking on the weekend would ensure that the reference repo size grows more slowly. 看到上述二分法后，一个自然的下一步就是提出：为什么不在周末进行全量重打包，而在工作日进行增量重打包呢？这似乎既能在工作日（消费者更活跃时）保证仓库的实时性，又能通过周末的全量重打包确保参考仓库的体积增长得更慢。

For simplicity, let’s say we’re writing the reference repos with some key <myrepo>-<timestamp>.tar into some bucket. To keep the size of the incrementally packed repo smaller, one can either: 为简单起见，假设我们将参考仓库以 <myrepo>-<timestamp>.tar 的键名写入某个存储桶。为了保持增量重打包仓库的体积较小，可以采取以下两种方式之一：

Write the wholesale repacked repos using the same key scheme, and have the incremental repacking jobs bootstrap from the last successful repack (wholesale or incremental), instead of from a clone operation. 使用相同的键名方案写入全量重打包的仓库，并让增量重打包任务从上一次成功的重打包（无论是全量还是增量）中引导（bootstrap），而不是从 clone 操作开始。
Write the wholesale repacked repos using an altered key scheme, say <myrepo>-<timestamp>-packed.tar, and bootstrap the incremental repack from… 使用修改后的键名方案写入全量重打包的仓库，例如 <myrepo>-<timestamp>-packed.tar，并从……引导增量重打包。