A fast and secure tar parser for the Rust ecosystem

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Starting with 0.131, uv uses tar-codec to parse source distributions. tar-codec is roughly twice as fast as our previous tar implementation (astral-tokio-tar) while also reducing uv’s exposure to parser differentials and other security risks. This blog post is about tar-codec itself: what it is, and what compelled us to build and optimize it. Read on if you, too, would like to know too much about tar archives.

从 0.131 版本开始，uv 使用 tar-codec 来解析源代码分发包。tar-codec 的速度大约是我们之前使用的 tar 实现（astral-tokio-tar）的两倍，同时也降低了 uv 在解析器差异和其他安全风险方面的暴露程度。这篇博文将介绍 tar-codec 本身：它是什么，以及是什么促使我们构建并优化它。如果你也想了解关于 tar 归档文件的过多细节，请继续阅读。

Background: uv and tar # What follows is an extended digression on the tar format. If all you care about is tar-codec, skip to it. As a Python packaging tool, uv is in the business of resolving and installing Python packages. uv requests installable distributions from various sources (principally package indices like PyPI) and installs them on the user’s behalf. This is how you go from uv add urllib3 to something you can actually import and use in your own Python codebase.

背景：uv 与 tar # 下面是对 tar 格式的延伸探讨。如果你只关心 tar-codec，可以直接跳过。作为一种 Python 打包工具，uv 的核心业务是解析和安装 Python 包。uv 从各种来源（主要是像 PyPI 这样的包索引）请求可安装的分发包，并代表用户进行安装。这就是你如何从执行 uv add urllib3 到获得可以在自己的 Python 代码库中实际导入和使用的内容的过程。

Installable distributions come in two primary forms: “Built” distributions, also known as bdists or (most commonly) wheels, are essentially ZIP archives containing structured metadata and a fixed layout. Installers handle these by (more or less) plopping their contents into the right destination directory. Many people think that bdist is short for “binary distribution,” since wheels often contain compiled native code. This one common way to use wheels, but wheels can also contain pure Python. They’re just another (and frequently better) way to package and install a distribution.

可安装的分发包主要有两种形式：“构建”分发包，也称为 bdist 或（最常见的）wheel，本质上是包含结构化元数据和固定布局的 ZIP 归档文件。安装程序通过（或多或少地）将它们的内容放入正确的目标目录来处理这些文件。许多人认为 bdist 是“二进制分发”的缩写，因为 wheel 通常包含编译后的原生代码。这确实是使用 wheel 的一种常见方式，但 wheel 也可以包含纯 Python 代码。它们只是打包和安装分发包的另一种（且通常更好）方式。

“Source” distributions, also known as sdists, are essentially tar archives. Unlike built distributions, a source distribution is essentially just a rollup of a source tree’s state at release time. By definition, source distributions execute arbitrary code at installation time. This makes them very powerful, but brittle (missing build tools or dependencies) and also a security risk (an attacker who controls the sdist can run code without the user needing to import anything). Wheels are generally produced by running a build backend on a source distribution. Examples of that include uv’s own uv_build, maturin, and flit.

“源代码”分发包，也称为 sdist，本质上是 tar 归档文件。与构建分发包不同，源代码分发包本质上只是发布时源代码树状态的汇总。根据定义，源代码分发包在安装时会执行任意代码。这使得它们非常强大，但也非常脆弱（缺少构建工具或依赖项），并且存在安全风险（控制 sdist 的攻击者可以在用户无需导入任何内容的情况下运行代码）。Wheel 通常是通过在源代码分发包上运行构建后端来生成的。其中的例子包括 uv 自己的 uv_build、maturin 和 flit。

Consequently, at a bare minimum, a Python package installer needs to understand at least two different archive formats: ZIP for wheels, and tar for sdists. We’ve already talked a bit before about ZIPs on the Astral blog, and we’ll certainly be talking about them again soon enough. But today we’ll be focusing on tar.

因此，作为最低限度，Python 包安装程序至少需要理解两种不同的归档格式：用于 wheel 的 ZIP 和用于 sdist 的 tar。我们之前已经在 Astral 博客上讨论过 ZIP，而且我们很快肯定会再次讨论它们。但今天我们将专注于 tar。

Why a new tar parser? # It’s fair to ask: why does uv want a new tar parser? In short: security and performance, but mostly security. Up until 0.13, uv used astral-tokio-tar for its tar operations, including both parsing and creation (in uv build). astral-tokio-tar is itself a fork of edera-dev/tokio-tar, which is a fork of vorot93/tokio-tar, which is a fork of dignifiedquire/async-tar, which is an async adaptation of composefs/tar-rs.

为什么需要一个新的 tar 解析器？# 问这个问题很合理：为什么 uv 需要一个新的 tar 解析器？简而言之：安全性和性能，但主要是安全性。在 0.13 版本之前，uv 使用 astral-tokio-tar 进行 tar 操作，包括解析和创建（在 uv build 中）。astral-tokio-tar 本身是 edera-dev/tokio-tar 的一个分支，后者是 vorot93/tokio-tar 的分支，而 vorot93/tokio-tar 又是 dignifiedquire/async-tar 的分支，后者则是 composefs/tar-rs 的异步适配版本。

Around a year ago, we began to receive a number of security reports in astral-tokio-tar. This spurred us into looking more into our compression and archive handling stack, where we found additional problems. Many of these boiled down to differentials that could cause a full parser desynchronization: given an identical tarball, uv, pip, and other package tooling could be made to see different contents for the same package distribution.

大约一年前，我们开始收到关于 astral-tokio-tar 的一些安全报告。这促使我们更深入地研究我们的压缩和归档处理栈，我们在其中发现了更多问题。其中许多问题归结为可能导致完全解析器不同步的差异：给定一个相同的 tarball，uv、pip 和其他包工具可能会被诱导为同一个包分发看到不同的内容。

We found this particularly concerning in the context of archives that would appear “safe” to an index (like PyPI) but could extract dangerously when parsed by an installer like uv or pip. This could the take the form of an “append” differential (where the attacker can append new files or new contents to existing files for a specific victim parser), or even a “full” differential (where the attacker can fully separate the extraction contents for each parser).

在那些对索引（如 PyPI）看起来“安全”，但当被像 uv 或 pip 这样的安装程序解析时可能危险地提取的归档文件背景下，我们发现这一点尤其令人担忧。这可能表现为“追加”差异（攻击者可以为特定的受害者解析器向现有文件追加新文件或新内容），甚至表现为“完全”差异（攻击者可以完全分离每个解析器的提取内容）。

In the context of uv, the attacker model for these kinds of vulnerabilities is somewhat subtle: tar is only used for source distributions which, per above and unlike wheels, are assumed to execute arbitrary code at build time. Consequently, Python packaging mostly considers tar differentials to be a minor concern, since any attacker who would use them has a much more direct and official route towards writing arbitrary contents to disk (or executing arbitrary code).

在 uv 的背景下，这类漏洞的攻击模型有些微妙：tar 仅用于源代码分发包，根据上述内容，与 wheel 不同，源代码分发包被假定在构建时执行任意代码。因此，Python 打包界大多认为 tar 差异是一个次要问题，因为任何会利用它们的攻击者都有更直接、更官方的途径向磁盘写入任意内容（或执行任意代码）。

Still, these findings unnerved us, and we embarked on a project to eliminate every differential we could find in astral-tokio-tar. tar-codec grew directly out of that effort: we ultimately came to the conclusion that the tokio-tar family was too lenient by design, and that retrofitting constraints to it would be more troublesome and error-prone (including for existing non-Astral users of its APIs) than developing our own, ground-up implementation of tar.

尽管如此，这些发现还是让我们感到不安，我们着手开展了一个项目，旨在消除我们在 astral-tokio-tar 中能找到的每一个差异。tar-codec 正是直接源于这一努力：我们最终得出结论，tokio-tar 系列在设计上过于宽松，对其进行约束改造比从头开始开发我们自己的 tar 实现更麻烦且更容易出错（包括对于其 API 的现有非 Astral 用户而言）。

This would also give us the opportunity to systematically understand the tar format, and ensure that our assumptions around which aspects are (or aren’t) differential risks hold true. Similarly, we determined that our performance goals around tar parsing would be impossible to meet in the existing code base, at least not without refactors that would be tantamount to a rewrite.

这也让我们有机会系统地理解 tar 格式，并确保我们关于哪些方面是（或不是）差异风险的假设是成立的。同样，我们确定，在现有的代码库中，我们关于 tar 解析的性能目标是不可能实现的，至少在不进行等同于重写的重构的情况下是无法实现的。

(t)archaeology # It’s important to understand that one does not simply build a tar parser from scratch. One must first understand how the tar format evolved into its current state. tar is an interesting format. Or more precisely, it’s an interesting family of formats. To summarize, there are three (and a half) variants that matter in practice: There’s UNIX V7 tar, which is the “original” tar. Very few tar implementations write this format by default anymore, but virtually all still read it. For example, Python’s tarfile module will happily parse V7 tar files. The UNIX V7 format is not widely used anymore because it comes with significant limitations on per-member file sizes (approximately 8 GiB) and path lengths (100 bytes). Then there’s the UStar/pax tar family. This family is widely supported by tar implementa

(t)考古学 # 必须明白，人们不会简单地从零开始构建一个 tar 解析器。首先必须了解 tar 格式是如何演变成当前状态的。tar 是一种有趣的格式。或者更准确地说，它是一个有趣的格式家族。总结一下，在实践中有三种（半）变体很重要：首先是 UNIX V7 tar，它是“原始”的 tar。现在很少有 tar 实现默认写入这种格式，但几乎所有实现仍然读取它。例如，Python 的 tarfile 模块可以愉快地解析 V7 tar 文件。UNIX V7 格式不再被广泛使用，因为它在每个成员文件大小（约 8 GiB）和路径长度（100 字节）方面存在重大限制。然后是 UStar/pax tar 家族。这个家族得到了 tar 实现的广泛支持。