Automated Data Readiness for Scientific AI

面向科学人工智能的自动化数据就绪化

Abstract: Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment.

摘要： 领先的计算设施管理着大规模的科学数据集，这些数据在作为人工智能训练数据使用前，通常需要进行大量的转换。然而，目前尚无现有的框架能够完全统一自动化转换、就绪性评估、溯源追踪以及智能体原生部署。

We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication.

我们提出了 REDI，这是一个开源框架，通过统一的五阶段流水线（摄取、预处理、转换、结构化和输出）填补了这一空白。该框架在每个阶段都配备了用于可重复性的监测工具，并可作为智能体可调用的技能进行部署；配套工具 SetGo 则实现了 FAIR 原则合规性和目录发布的自动化。

Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case.

通过在气候学、蛋白质组学、材料科学和核聚变领域的评估，REDI 将所有数据集从原始状态转换为 AI 就绪状态，其输出结果已通过领域专家参考资料的验证。初步结果显示，在气候学案例中，该框架在 Frontier 超级计算机上实现了近乎理想的 100 节点并行扩展。

Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.

基于溯源的性能分析显示，文件 I/O 是流水线的主要成本来源，而格式选择则是首要的优化手段。这些结果确立了 REDI 作为跨领域平台的地位，它为科学人工智能提供了自动化的数据就绪能力，将数据准备的瓶颈转化为可重复、可复用的社区资产。