DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
DataPrep-Bench：大语言模型作为训练数据准备器的基准测试
Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. 摘要： 训练数据的质量从根本上决定了大语言模型（LLM）的能力，然而目前尚无统一的基准来衡量 LLM、智能体（Agents）以及以数据为中心的工作流在端到端准备训练数据方面的实际表现。
We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, “quality” refers to downstream training utility rather than surface-level textual properties. 我们将 LLM 驱动的数据准备视为包含两个互补的能力：一是“数据构建”，即将原始资源转换为监督训练数据；二是“数据质量评估”，即在下游训练之前预测候选数据集的训练价值。在此过程中，“质量”指的是下游训练的效用，而非表层的文本属性。
We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. 我们推出了 DataPrep-Bench，这是首个统一的基准测试，旨在通过共享的、基于下游任务的协议，在六个领域和多个基础模型上联合评估上述两种能力。
For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. 在数据构建方面，各种方法使用相同的原始资源，并通过将基础模型在这些输出与 Dolly-15k 联合微调后的表现进行评分。与此同时，我们发布了 Data-Construction-Skill，这是一个技能引导型智能体，它在 Llama-3.1-8B 金融领域将仅使用 Dolly 的基线提升了近 20 个绝对百分点，并在知识提取密集型领域与最强的基于智能体和 DataFlow 的方法具有竞争力。
For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. 在数据质量评估方面，评分函数通过与共享候选池中下游性能的皮尔逊相关系数（Pearson correlation）进行评分；我们发布了分布对齐分数（DAS），这是一种基于分布的评估器，利用候选数据集与领域代理（domain proxy）之间的最大均值差异（MMD）进行计算。
DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DAS 在六个领域中的四个领域实现了最强的跨模型相关性，并且是唯一一个在数学、科学和医学领域同时达到 r > 0.70 的指标，优于现有的基于质量、多样性和启发式的评估器。
DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation. DataPrep-Bench 提供了一个统一的、基于下游任务的框架，用于衡量作为 LLM 驱动数据准备中同等重要的两个目标的进展。