PRX Part 4: Our Data Strategy

PRX 第四部分：我们的数据策略

Welcome back! This is Part 4 of the PRX series. Parts 1 to 3 covered model architectures, training design, and a 24-hour speedrun. This time we’re pulling back the curtain on the part that quietly underpins all of it: the data. 欢迎回来！这是 PRX 系列的第四部分。前三部分涵盖了模型架构、训练设计以及 24 小时极速训练。这一次，我们将揭开支撑这一切的幕后功臣：数据。

Of all the things that shaped PRX’s quality, the data pipeline was one of the least glamorous parts to build but nevertheless an important piece to get right. Here’s what we did, what we’d do differently, and a few things we only learned the slow way. 在所有塑造 PRX 质量的因素中，数据流水线是最不引人注目的部分之一，但它依然是必须做好的关键环节。以下是我们所做的工作、我们打算改进的地方，以及一些我们通过“慢功夫”才学到的经验。

In one sentence: we assemble training data from a mix of public and internal datasets, re-caption the images with a VLM, and turn the result into the streamable corpus we trained PRX on. At a high level, the data pipeline looks like this: In the following we will dive into it in detail. 简单来说：我们从公共和内部数据集中汇编训练数据，利用视觉语言模型（VLM）为图像重新生成标题，并将结果转化为我们训练 PRX 所需的可流式传输语料库。从宏观上看，数据流水线如下所示：接下来我们将深入探讨细节。

1. Guiding principles

1. 指导原则

A diverse dataset for pre-training 用于预训练的多样化数据集

The goal was to assemble a large, diverse dataset for pre-training. At this stage the model is learning how the world looks: the visual concepts, the objects and scenes, how things are composed and lit, and the sheer range of what images can contain. That is a problem of coverage and diversity, not of per-image perfection. 我们的目标是汇编一个庞大且多样化的预训练数据集。在这个阶段，模型正在学习世界的样子：视觉概念、物体与场景、事物的构成与光影，以及图像所能包含的广阔范围。这是一个关于覆盖面和多样性的问题，而非追求单张图像的完美。

A broad, representative corpus teaches the model far more about the structure of the visual world than a smaller, prettier one would, even if many of the individual images are ordinary snapshots or slightly compressed. Over-filtering for aesthetics at this stage would actually hurt, narrowing the distribution and costing the model concepts and compositional variety it cannot recover later. 一个广泛且具有代表性的语料库，比一个规模较小但更“精美”的语料库能让模型学到更多关于视觉世界结构的内容，即使其中许多图像只是普通的快照或经过轻微压缩。在这个阶段过度筛选美学质量反而有害，这会缩小数据分布，导致模型丢失后续无法弥补的概念和构图多样性。

Making generations look polished is a separate, later concern, which we leave to fine-tuning and preference alignment on small, ruthlessly curated sets. Pre-training is for breadth; fine-tuning is for taste. 让生成结果看起来精致是后续需要考虑的问题，我们将其留给在小规模、经过严格筛选的数据集上进行的微调和偏好对齐。预训练是为了广度，微调则是为了品味。

A mix of data sources 数据源的混合

We assemble our pre-training data from a mix of public and internal datasets. The priority at this stage is breadth, diversity, and standing on curation that already exists rather than redoing it ourselves. Where a source already comes quality-filtered, deduplicated, and filtered for NSFW content and personal information, we build on that work instead of repeating it at scale. 我们通过混合公共和内部数据集来汇编预训练数据。现阶段的优先级是广度、多样性，并利用现有的策展成果，而不是重复造轮子。如果某个数据源已经过质量过滤、去重，并剔除了 NSFW 内容和个人信息，我们就会直接利用这些成果，而不是大规模地重复劳动。

Sources arrive in different shapes: some come with the image data itself, others as metadata plus baseline captions that we bring into a common form. We took a pragmatic approach: rather than build a corpus entirely from scratch, we leaned on existing datasets and our own tooling to assemble one quickly. In hindsight it is not necessarily the absolute best dataset one could build, but it was a solid and lightweight starting point for pre-training a 7B model. 数据源的形式各异：有些自带图像数据，有些则包含元数据和基础标题，我们需要将其统一为通用格式。我们采取了务实的方法：与其从零开始构建语料库，不如依靠现有数据集和我们自己的工具快速组装。事后看来，这未必是能构建出的绝对最好的数据集，但它为预训练 7B 模型提供了一个稳健且轻量级的起点。

Our captions philosophy 我们的标题哲学

In our experience, what matters most for pre-training is to use long captions that accurately describe everything in the image. We saw this directly in Part 2, where switching from short captions to long ones substantially improved sample quality. If captioning is faithful, we don’t need to worry about the occasional screenshot, advertisement, logo, or bit of text in an image, because those things get described in the caption too, so the model learns them as conditioned, controllable attributes rather than reproducing them unconditionally. 根据我们的经验，预训练中最重要的是使用能准确描述图像中所有内容的详细标题。我们在第二部分中直接观察到了这一点：从短标题切换到长标题后，样本质量得到了显著提升。如果标题描述准确，我们就不必担心图像中偶尔出现的截图、广告、Logo 或文字，因为这些内容也会被描述在标题中，模型会将它们作为有条件的、可控的属性来学习，而不是无条件地复现它们。

Accurate captioning turns “noise” into something you can prompt for, or prompt away. This is precisely why the filtering we do later is deliberately light. We remove what is genuinely unusable, not everything that is imperfect. 准确的标题将“噪声”变成了你可以通过提示词来调用或剔除的内容。这正是我们后续过滤工作刻意保持“轻量”的原因。我们只剔除真正无法使用的内容，而不是剔除所有不完美的内容。

Data formats 数据格式

We have been using Mosaic Streaming and Mosaic Data Shards (MDS) as a dataset format for distributed training for a while now. In combination with Mosaic Composer we found it to be a very low-maintenance, flexible, and well-performing framework for distributed training. Furthermore, MDS datasets can easily and effectively be mixed and shuffled and also allow for distributed training directly from object storage like S3 or GCS. 我们使用 Mosaic Streaming 和 Mosaic Data Shards (MDS) 作为分布式训练的数据集格式已经有一段时间了。结合 Mosaic Composer，我们发现这是一个维护成本极低、灵活且性能优异的分布式训练框架。此外，MDS 数据集可以轻松有效地进行混合和洗牌，并支持直接从 S3 或 GCS 等对象存储中进行分布式训练。

However, MDS datasets are very rigid. Adding a column or creating a subset for a given filter basically means that one has to scan and rewrite the whole dataset. That’s why we use Lance for this kind of feature engineering and dataset curation. Lance is a columnar data format with cheap predicate pushdown, scalar indexes, and vector search, the right tool for building and exploring datasets with billions of rows. These two formats play off each other throughout this post and the PRX data pipeline: Lance to build, MDS to stream. 然而，MDS 数据集非常僵化。添加一列或为特定过滤器创建子集，基本上意味着必须扫描并重写整个数据集。这就是为什么我们使用 Lance 进行此类特征工程和数据集策展。Lance 是一种列式数据格式，具有低成本谓词下推、标量索引和向量搜索功能，是构建和探索数十亿行数据集的理想工具。这两种格式在本文和 PRX 数据流水线中相辅相成：用 Lance 构建，用 MDS 流式传输。

On text latents 关于文本潜变量（Text Latents）

For previous training runs, we used T5Gemma as the text encoder and pre-computed the text latents, storing them in MDS as bytes. This time around, after switching the text encoder to Qwen3-VL, we decided to compute text latents on the fly during training instead. Running the text encoder inside the training loop costs throughput, but how much depends on the model: for a small denoiser it can be significant, whereas at PRX’s 7B scale the text encoder’s compute is negligible next to the denoiser, and we measured only a roughly 3–4% throughput cost (about 1 extra day on a 30-day run). 在之前的训练中，我们使用 T5Gemma 作为文本编码器，并预先计算文本潜变量，将其以字节形式存储在 MDS 中。这一次，在将文本编码器切换为 Qwen3-VL 后，我们决定在训练过程中实时计算文本潜变量。在训练循环中运行文本编码器会消耗吞吐量，但具体影响取决于模型：对于小型去噪器，这种影响可能很显著；而在 PRX 的 7B 规模下，文本编码器的计算量相对于去噪器来说微不足道，我们测得的吞吐量损耗仅为 3-4% 左右（在 30 天的训练周期中大约多出 1 天）。

In return we get two things. Skipping pre-computation keeps our MDS shards much smaller, small enough to store the full pre-training dataset on the SSD-backed shared filesystem of our SLURM cluster rather than streaming it over the network from object storage. And it keeps us free to change the text encoder later without rewriting terabytes of stored latents, which is exactly the kind of switch we made when moving to Qwen3-VL. 作为回报，我们获得了两点优势。跳过预计算使我们的 MDS 分片小得多，小到足以将整个预训练数据集存储在我们 SLURM 集群的 SSD 共享文件系统上，而无需通过网络从对象存储中流式传输。此外，这使我们能够自由地在后续更换文本编码器，而无需重写数 TB 的存储潜变量，这正是我们在转向 Qwen3-VL 时所做的切换。

On image encoding 关于图像编码

We encoded all images as JPEG at quality 92, instead of a lossless format such as PNG. We did not just assume quality 92 was safe, we measured it. Real-world images have usually already been JPEG-compressed several times, so the real question is whether one more re-encode hurts. Across repeated decode/encode cycles on both high-resolution… 我们将所有图像编码为质量为 92 的 JPEG 格式，而不是 PNG 等无损格式。我们并非仅仅假设质量 92 是安全的，而是进行了实际测量。现实世界中的图像通常已经被多次 JPEG 压缩，所以真正的问题在于再进行一次重编码是否会有损害。通过对高分辨率图像进行反复的解码/编码循环测试……