The Silent Failures in Agentic Data Science

Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of data science agents that promise automating the design and execution of data science from task formulation to model development.

大型语言模型显著降低了编写代码的门槛。自然地，这也促使了数据科学智能体的开发，这些智能体承诺能够实现从任务制定到模型开发的数据科学设计与执行的自动化。

Existing evaluations of agentic data science (agentic DS) assess downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental principles of data science: methodological data science- and domain knowledge.

现有的针对智能体数据科学（agentic DS）的评估主要考量下游预测性能和代码正确性，这为理解智能体数据科学的能力提供了初步见解。我们认为，这些评估忽略了数据科学的基本原则：方法论数据科学知识和领域知识。

As such, only assessing downstream performance masks methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles.

因此，仅评估下游性能掩盖了诸如数据泄露和不当验证等方法论上的缺陷，从而限制了稳健的智能体数据科学系统的发展。在本文中，我们从三个不同的角度对智能体数据科学进行了超越准确率的评估。