Inside Zig’s Incremental Compilation

深入了解 Zig 的增量编译

As a member of the Zig core team, one of the most impactful projects I’ve been involved with is the implementation of incremental compilation into the Zig compiler. This feature allows the compiler to detect which individual functions and declarations have changed since a project was last built, recompile only that code, and directly patch the resulting bytes into the output binary, making the rebuild extremely fast. 作为 Zig 核心团队的一员，我参与过的最具影响力的项目之一，就是为 Zig 编译器实现增量编译。该功能允许编译器检测自上次构建以来哪些具体的函数和声明发生了变化，仅重新编译这些代码，并将生成的字节直接修补到输出的二进制文件中，从而使重建过程变得极快。

The Zig project has been working towards this feature for a long time, and over the last few release cycles, it has finally gone from a proof-of-concept quality feature to one which is viable for real-world projects and which most of the Zig core team makes daily use of. Zig 项目长期以来一直致力于实现这一功能。在过去的几个发布周期中，它终于从一个概念验证阶段的功能，演变为适用于实际项目、且大多数 Zig 核心团队成员都在日常使用的成熟特性。

Today, using Zig’s incremental compilation, you can make changes to real, complex applications in a matter of milliseconds. But don’t just take my word for it! Here’s a simple video (no audio) demonstrating me using Zig to quickly make and test some changes to Fizzy, a pixel editor application. The initial build takes around 5 seconds, and then every time I make a change, a rebuild completes in 50–70ms. 如今，利用 Zig 的增量编译，你可以在几毫秒内对真实、复杂的应用程序进行修改。但别只听我的一面之词！这里有一个简单的视频（无音频），演示了我如何使用 Zig 快速对像素编辑器应用 Fizzy 进行修改和测试。初始构建大约需要 5 秒，而之后每次我进行修改时，重建仅需 50–70 毫秒即可完成。

For this demo, I had to upgrade Fizzy to Zig’s master branch. This is because while Zig 0.16.0 does have support for incremental compilation, it is missing some important linker features which have since been implemented. This means that if you prefer to stick to tagged releases of Zig, you likely won’t be able to try this out until 0.17.0 drops; sorry! 为了进行演示，我不得不将 Fizzy 升级到 Zig 的 master 分支。这是因为虽然 Zig 0.16.0 确实支持增量编译，但它缺少了一些后来才实现的重要链接器功能。这意味着如果你倾向于使用 Zig 的稳定版本（tagged releases），那么在 0.17.0 发布之前，你可能无法尝试此功能；抱歉！

If you’re already convinced and just want to know how to use this, great! Head on down to the last section of this post to find out. But perhaps you’re understandably skeptical that this is applicable to most projects, or, like me, you just enjoy learning how stuff like this works. For all of you folks, let’s dig into the details! 如果你已经被说服并只想知道如何使用它，那太好了！请直接跳转到本文的最后一部分了解详情。但也许你对它是否适用于大多数项目持怀疑态度，或者像我一样，只是喜欢研究这类技术的工作原理。对于你们所有人，让我们深入探讨一下细节吧！

Processing Source Files

处理源文件

The Zig compiler’s pipeline can be split up into a few different parts, which we’ll look at in order. The first part works at the granularity of entire source files, and basically consists of running the following process in a loop: Zig 编译器的流水线可以分为几个不同的部分，我们将按顺序查看。第一部分以整个源文件为粒度进行工作，基本上由以下循环过程组成：

1. Read in a source file from disk
2. Parse that file into an AST
3. Convert that AST into a format named “ZIR” using a pass named “AstGen”
4. 从磁盘读取源文件
5. 将该文件解析为抽象语法树（AST）
6. 使用名为“AstGen”的通道将该 AST 转换为名为“ZIR”的格式

If you’re curious, ZIR (Zig Intermediate Representation) is an untyped SSA-form IR—but don’t worry if you have no idea what that means, because it won’t really matter here. All we care about is that we’re converting an entire source file into a different format. 如果你感到好奇，ZIR（Zig 中间表示）是一种无类型的 SSA 形式 IR——但如果你不知道这意味着什么也不必担心，因为在这里它并不重要。我们唯一关心的是，我们正在将整个源文件转换为另一种格式。

While AstGen runs, it learns about all Zig imports (@import(“foo.zig”)) in the source file, so we can repeat this entire process on all of the imported files. So by running this process in a loop, we will ultimately discover every Zig source file in the compilation, and will convert them all to ZIR. 当 AstGen 运行时，它会识别源文件中的所有 Zig 导入（@import(“foo.zig”)），因此我们可以在所有导入的文件上重复整个过程。通过循环运行此过程，我们最终会发现编译中的每一个 Zig 源文件，并将它们全部转换为 ZIR。

This part of the pipeline actually has several useful properties: 流水线的这一部分实际上具有几个有用的特性：

• The processing run on each file is a pure function of that file’s contents, involving no shared or external state.
• Parse and AstGen are both quite fast on their own: on my laptop, running them both over the entire src/ directory of the Zig compiler (with no parallelism at all) takes around 920ms.
• Thanks to Zig’s usage of data-oriented design patterns, ZIR can be trivially written to and read from disk with one writev/readv system call—there is no “serialization” step.
• 每个文件的处理过程都是其文件内容的纯函数，不涉及任何共享或外部状态。
• Parse 和 AstGen 本身都非常快：在我的笔记本电脑上，对 Zig 编译器整个 src/ 目录运行这两者（完全不使用并行）大约需要 920 毫秒。
• 得益于 Zig 对面向数据设计模式的使用，ZIR 可以通过一次 writev/readv 系统调用轻松地写入磁盘或从磁盘读取——没有所谓的“序列化”步骤。

These properties have two nice consequences. Firstly, assuming one “task” per source file, this entire process is embarrassingly parallel. That means we can trivially run it on a thread pool by queuing up a task every time we discover a new source file from an import—the only shared state (which we’ll just protect with a mutex) is a hash set keeping track of which file paths we have already seen. 这些特性带来了两个很好的结果。首先，假设每个源文件是一个“任务”，整个过程是极易并行的。这意味着我们可以通过在每次从导入中发现新源文件时排队一个任务，轻松地在线程池上运行它——唯一的共享状态（我们将用互斥锁保护）是一个用于跟踪已处理文件路径的哈希集合。

Secondly, and arguably even more importantly, these properties make it very straightforward to implement incremental compilation for this part of the pipeline. All we need to do is cache each source file’s generated ZIR on disk, and only rebuild it when we detect that the file changed. 其次，可以说更重要的是，这些特性使得为流水线的这一部分实现增量编译变得非常简单。我们所要做的就是将每个源文件生成的 ZIR 缓存在磁盘上，并且仅在检测到文件发生更改时才重新构建它。

Both of these optimizations have been enabled by default in Zig for years—they are battle-tested and make this part of the pipeline near-instantaneous in most cases. If you’re using Zig, you can see how fast this is using the progress output on stderr—when it says “AST Lowering”, this part of the pipeline is running. I’d guess that a lot of Zig users only even notice that happening the very first time they run the compiler (because on its first run the compiler needs to do this work for the entire Zig standard library and compiler_rt). 这两种优化在 Zig 中已经默认开启多年——它们经过了实战检验，使得流水线的这一部分在大多数情况下几乎是瞬时的。如果你正在使用 Zig，可以通过 stderr 上的进度输出看到它的速度——当显示“AST Lowering”时，流水线的这一部分正在运行。我猜很多 Zig 用户只有在第一次运行编译器时才会注意到这一点（因为在第一次运行时，编译器需要为整个 Zig 标准库和 compiler_rt 执行此工作）。

Okay, so, we made this part fast! That’s great, but the bad news is that this was the easy part—lots of compilers can already do this kind of caching. From here, things will get trickier. 好了，我们让这一部分变快了！这很好，但坏消息是这只是简单部分——许多编译器已经可以做到这种缓存。从这里开始，事情会变得更加棘手。

Semantic Analysis

语义分析

The next part of the pipeline is arguably the most important: semantic analysis. This includes both type checking and comptime evaluation. 流水线的下一部分可以说是最重要的：语义分析。这包括类型检查和编译时（comptime）求值。

The job of semantic analysis is essentially to “interpret” the ZIR we produced earlier, emitting compile errors (such as type errors) along the way; and, for runtime functions, building another intermediate representation which can be sent on to later parts of the pipeline. 语义分析的工作本质上是“解释”我们之前生成的 ZIR，在此过程中发出编译错误（如类型错误）；对于运行时函数，则构建另一个可以发送到流水线后续部分的中间表示。

Before we move forward, a quick terminology clarification. A “container-level declaration” is the Zig equivalent of what other languages call a “top-level declaration”. That term is inaccurate in Zig, because container-level declarations do not have to be at the top level syntactically, but the concept is the same. If I say “container-level declaration”, I basically mean “a function, global constant, or global variable”. 在继续之前，先澄清一下术语。“容器级声明”（container-level declaration）是 Zig 中对应于其他语言所谓的“顶层声明”的概念。在 Zig 中，“顶层”这个词并不准确，因为容器级声明在语法上不必位于顶层，但概念是一样的。如果我说“容器级声明”，我基本上是指“函数、全局常量或全局变量”。

Semantic analysis is the most difficult part of the compiler to handle incrementally. Perhaps unsurprisingly then, this is where language design starts to matter a lot: while I am pretty confident that most modern languages could support incremental compilation similar to how we do, certain design decisions can make that much more difficult. Zig has had its design tweaked… 语义分析是编译器中最难进行增量处理的部分。因此，语言设计在这里开始变得非常重要也就不足为奇了：虽然我非常有信心大多数现代语言都能支持类似于我们这种方式的增量编译，但某些设计决策会使其变得困难得多。Zig 的设计已经经过了调整……