Building (systems) software with Nix

使用 Nix 构建（系统）软件

Something I found incredibly frustrating when I started programming was that I could not spend more time building projects and instead had to fight with errors that seemed like a distraction. These major time sinks didn’t seem more than an inconvenience that were preventing me from making progress and focus on the fun stuff. Even worse, pretty often what used to work would stop to and it would not be clear why. At the time, I didn’t have the knowledge to even hypothesise what might have happened.

当我刚开始编程时，最让我沮丧的事情莫过于无法将更多时间花在构建项目上，反而不得不去处理那些看似干扰项的错误。这些巨大的时间黑洞不仅仅是麻烦，它们阻碍了我的进度，让我无法专注于真正有趣的事情。更糟糕的是，很多时候原本能运行的代码突然就失效了，而且原因不明。当时，我甚至没有足够的知识去推测到底发生了什么。

Years later, and despite having levelled up as an engineer, I suppose I tend to feel something similar, especially when working on systems software. The project that built or run yesterday is no longer doing so in the same way. Nowadays I can come up with some possible reasons for this, test some theories, and so on, which is neat, but often this is a big time investment, and more importantly, it’s work that I don’t find too enjoyable and that takes time from what I want to do: actually building software. The fun stuff. There you have it, another moving target.

多年以后，尽管我的工程师水平有所提升，但我发现自己在处理系统软件时，依然会有类似的感受。昨天还能构建或运行的项目，今天却无法以同样的方式运行了。如今，我可以列举出一些可能的原因并进行验证，这固然不错，但这往往需要投入大量时间。更重要的是，我不觉得这项工作有多愉快，它占用了我真正想做的事情——即构建软件——的时间。这就是所谓的“移动目标”（moving target）。

Most modern programming languages come with package managers (bundler, cargo, npm, etc) that help fetch dependencies, and among other things they do their best to ensure that they aren’t altered without the user knowing. This is essential for security, and also helps to keep consistency across machines. This is a big improvement from how most people were building software not that long ago. This is not bullet-proof, though, since there are other sources of variability. The compiler or interpreter version, system-provided libraries, environment variables, etc.

大多数现代编程语言都配备了包管理器（如 bundler、cargo、npm 等），它们不仅能帮助获取依赖项，还能尽力确保这些依赖项不会在用户不知情的情况下被篡改。这对安全性至关重要，也有助于保持不同机器间的一致性。与不久前大多数人构建软件的方式相比，这已经有了巨大的进步。然而，这并非万无一失，因为还存在其他变数，例如编译器或解释器版本、系统提供的库、环境变量等。

If this weren’t enough, when dealing with systems-level software, we might be using native libraries, and exercise the operating system in ways most other software might never do! The more surface area, the higher the chances of the software we depend changing their behaviour over time. What about our operating system kernel? From an abstract perspective, it’s another library that code interfaces with. This leads us to some of the fastest evolving parts in the Linux kernel: the BPF subsystem, the fancy new sched_ext schedulers, or io_uring, among others.

如果这还不够，在处理系统级软件时，我们可能会使用原生库，并以大多数其他软件永远不会使用的方式调用操作系统！涉及的层面越多，我们所依赖的软件随时间改变其行为的可能性就越大。那么操作系统内核呢？从抽象的角度来看，它只是代码与之交互的另一个库。这让我们不得不面对 Linux 内核中演进最快的部分：BPF 子系统、新颖的 sched_ext 调度器或 io_uring 等。

When it comes to the whole operating system, a seemingly scheduled routine system update can update part of the toolchain such as the compiler or the linker. A system library that your code uses might change, or even the kernel itself! This can prevent your code from not building, or even worse: changing its behaviour.

对于整个操作系统而言，一次看似常规的系统更新可能会更新工具链的一部分，例如编译器或链接器。你代码所使用的系统库可能会发生变化，甚至内核本身也可能更新！这可能导致你的代码无法构建，或者更糟糕的是：改变其运行行为。

One of each, please

每样来一份

I haven’t checked the data but I would find surprising if the BPF ecosystem is not one of the most fast evolving ones in Linux. This is great, because it means that every kernel release comes with exciting new features and fixes, but it can add spiciness to our development (and production!) environments. Besides that, for a basic libbpf based application written in Rust, we depend on: Rust compiler (rustc) and related tools (cargo); C compiler(s) (clang). Typically BPF code gets compiled with clang, but most Linux systems use gcc for other native code; Native dependencies (libelf, zlib, libbpf, libc); Rust dependencies; Linux kernel headers (vmlinux.h), and, if generated at build time, the system’s BTF information and bpftool to produce the header file; Kernel-side of BPF tooling (system call layer, verifier, etc); Static linker(s); Dynamic loader (ld.so); The environment (env); If I were to bet on this list being complete, I would probably lose.

我没有查过数据，但如果 BPF 生态系统不是 Linux 中演进最快的生态系统之一，我会感到非常惊讶。这固然很好，因为这意味着每个内核版本都带来了令人兴奋的新功能和修复，但也给我们的开发（和生产！）环境增添了“调料”。此外，对于一个用 Rust 编写的基本 libbpf 应用程序，我们依赖于：Rust 编译器 (rustc) 及其相关工具 (cargo)；C 编译器 (clang)。通常 BPF 代码是用 clang 编译的，但大多数 Linux 系统使用 gcc 处理其他原生代码；原生依赖项 (libelf, zlib, libbpf, libc)；Rust 依赖项；Linux 内核头文件 (vmlinux.h)，如果是在构建时生成，还需要系统的 BTF 信息和 bpftool 来生成头文件；内核侧的 BPF 工具（系统调用层、验证器等）；静态链接器；动态加载器 (ld.so)；环境 (env)；如果我打赌这份清单是完整的，我大概率会输。

But hopefully it’s enough to illustrate the problem we are dealing with. We are a single apt/dnf/$YOUR_PACKAGE_MANAGER away from large parts of our software dependencies potentially changing. By default, it’s just one part of the equation that’s fully understandable and under our control: the Rust dependencies (assuming there’s a Cargo.lock in SCM). Perhaps also libbpf, since it’s vendored by default. If you are unlucky, the burden of hunting for differences is on you. More often that not, figuring out what the previous state of the system was is a herculean task.

但希望这足以说明我们正在处理的问题。我们距离软件依赖项的大规模潜在变更，仅仅隔着一个 apt/dnf/$YOUR_PACKAGE_MANAGER 命令。默认情况下，只有方程的一部分是完全可理解且受我们控制的：Rust 依赖项（假设源代码管理中有 Cargo.lock）。也许还包括 libbpf，因为它默认是作为 vendored 依赖存在的。如果你运气不好，寻找差异的重担就落在了你身上。通常情况下，弄清楚系统之前的状态是一项艰巨的任务。

Taming the dependency chaos

驯服依赖地狱

One of the core ideas behind modern dependency managers is the inclusion of lockfiles. They represent a serialised view of the dependencies that were “locked in” at some point. Among other data, this includes the package location, its precise version, and a checksum to ensure it’s not tampered with or corrupt. Wouldn’t it be neat if we could apply it for all the items in the list above?

现代依赖管理器背后的核心思想之一是引入锁文件（lockfiles）。它们代表了在某个时间点被“锁定”的依赖项的序列化视图。除其他数据外，这包括包的位置、精确版本以及确保其未被篡改或损坏的校验和。如果我们能将此应用于上述列表中的所有项目，那该多好？

Turns out there are more than one implementation of this idea, one them being Nix. Nix is: a package manager (provides libraries, binaries, etc); a build system (instructions on how to build code); a programming language used in the above; a Linux distribution, NixOS, based on the eponymous package manager and build system; a bunch of infrastructure such as a build farm, cache, etc to support the above;

事实证明，实现这一想法的方案不止一种，Nix 就是其中之一。Nix 是：一个包管理器（提供库、二进制文件等）；一个构建系统（关于如何构建代码的指令）；一种用于上述用途的编程语言；一个基于同名包管理器和构建系统的 Linux 发行版 NixOS；以及一系列支持上述功能的基础设施，如构建农场、缓存等。

Here we are referring to the package manager and build system bits, which work in most Linux distributions and other operating systems such as MacOS. In fact, while I have been using these for almost 2 years now, I am a new NixOS user. There is a lot of great content online by folks that know way more about Nix than I do on the language, developer environments, how Nix works under the hood, etc; For a simplified mental model, the pieces that are useful to know are that the “official” Nix packages are in this repo called nixpkgs, and that if we want to have something like a Cargo.lock for Nix, we probably want to use flakes. By using flakes we can select a particular revision of the nixpkgs repository and every dependency we use will match what’s defined and built in it.

这里我们指的是包管理器和构建系统部分，它们可以在大多数 Linux 发行版和其他操作系统（如 MacOS）上运行。事实上，虽然我已经使用这些工具近两年了，但我还是 NixOS 的新手。网上有很多由比我更了解 Nix 的人撰写的优质内容，涵盖了语言、开发环境、Nix 底层工作原理等。为了建立一个简化的思维模型，你需要知道的是：“官方” Nix 包位于名为 nixpkgs 的仓库中；如果我们想为 Nix 拥有类似 Cargo.lock 的功能，我们可能需要使用 flakes。通过使用 flakes，我们可以选择 nixpkgs 仓库的特定修订版本，我们使用的每个依赖项都将与其中定义和构建的内容相匹配。

A Rust + BPF environment using Nix

使用 Nix 构建 Rust + BPF 环境

See runqslower, with a Nix flake similar to the one that I use. I commented it enough to make it more understandable to newcomers, but tried not to be too verbose. Once Nix in installed, a simple nix develop brings an environment with all the required software. The best part is that it’s of

请参考 runqslower，其中包含一个与我使用的类似的 Nix flake。我添加了足够的注释以方便新手理解，同时尽量保持简洁。一旦安装了 Nix，只需运行 nix develop 即可获得包含所有所需软件的环境。最棒的部分是……