The Problem with pandas Isn’t Performance. It’s Cognitive Overhead.

pandas 的问题不在于性能，而在于认知负担。

Faster dataframe engines are nice, but they don’t reduce the amount of syntax an analyst has to hold in their head. 更快的 DataFrame 引擎固然很好，但它们并没有减少分析师脑海中需要记忆的语法量。

For years, the conversation about pandas has focused on performance. As its creator has acknowledged, pandas’ foundations were not built for today’s data workloads. Over time, pandas has made real progress in this area, particularly with the recent pandas 3.0 release. Meanwhile, Polars and DuckDB have shown what can be achieved when modern data structures are part of the design from the beginning. 多年来，关于 pandas 的讨论一直集中在性能上。正如其创始人所承认的那样，pandas 的基础并非为当今的数据工作负载而构建。随着时间的推移，pandas 在这一领域取得了实质性进展，特别是在最近发布的 pandas 3.0 中。与此同时，Polars 和 DuckDB 展示了当现代数据结构从设计之初就融入其中时，能够实现怎样的效果。

But performance is only one cost in data analysis. For many everyday tasks, back-end performance is a secondary concern. The dataset fits in memory. The calculation finishes in a second. But the analyst has to take time to remember an API, rearrange brackets, look up an aggregation pattern and check whether a grouping key has quietly become an index. The CPU is idle. The human is not. The deeper problem with pandas—and, to different degrees, with most dataframe APIs—is cognitive overhead. 但性能只是数据分析中的成本之一。对于许多日常任务而言，后端性能是次要的。数据集可以放入内存，计算在一秒内完成。但分析师必须花时间去记忆 API、调整括号、查找聚合模式，并检查分组键是否悄悄变成了索引。CPU 在闲置，而人类却在忙碌。pandas（以及大多数 DataFrame API）更深层次的问题在于认知负担。

The tax hidden in ordinary code

普通代码中隐藏的“税”

Consider a simple task: keep positive sales, calculate a margin, summarise by region and sort the result: 考虑一个简单的任务：保留正数销售额，计算利润率，按地区汇总并对结果进行排序：

summary = (
    sales.loc[sales["revenue"] > 0]
    .assign(margin=lambda df: df["revenue"] - df["cost"])
    .groupby("region", as_index=False)
    .agg(
        total_revenue=("revenue", "sum"),
        average_margin=("margin", "mean"),
    )
    .sort_values("total_revenue", ascending=False)
)

This is not bad pandas. It is not a deliberately awful example assembled to win a syntax comparison. An experienced pandas user can read it without difficulty. But notice how much of the expression is about negotiating with the API rather than detailing actual logic: 这不是糟糕的 pandas 代码，也不是为了在语法对比中获胜而故意拼凑的拙劣示例。经验丰富的 pandas 用户可以轻松阅读它。但请注意，该表达式中有多少部分是在与 API 进行“博弈”，而不是在描述实际逻辑：

A column is sometimes sales["revenue"], sometimes df["revenue"], and sometimes the string "revenue".
Creating a column requires assign and a lambda if we want to preserve the method chain.
A named aggregation is expressed as a tuple whose order is column first, function second.
Descending order is expressed by setting an ascending option to False.
The behaviour of the grouping key depends on as_index, a parameter whose significance is not obvious from the analytical task.
列有时是 sales["revenue"]，有时是 df["revenue"]，有时又是字符串 "revenue"。
如果我们想保持方法链，创建列就需要使用 assign 和 lambda。
命名聚合表示为一个元组，其顺序是先列名后函数。
降序是通过将 ascending 选项设置为 False 来表示的。
分组键的行为取决于 as_index，这个参数的重要性在分析任务中并不直观。

None of these details is individually difficult. But each one consumes a small piece of human working memory that could otherwise be used to think about data. pandas indexes are a good example of this tension. The familiar appearance of .reset_index() after a group-by is not just a few extra keystrokes; it is an annoying distraction. And of course, backward compatibility limits how radically a mature library can redesign its surface. 这些细节单独来看都不难，但每一个都会消耗人类工作记忆的一小部分，而这些记忆本可以用来思考数据。pandas 索引就是这种矛盾的一个很好的例子。在分组后频繁出现的 .reset_index() 不仅仅是多敲了几下键盘，它更是一种令人烦恼的干扰。当然，向后兼容性也限制了一个成熟的库对其界面进行彻底重新设计的能力。

“AI can write it now” is only half an answer

“现在 AI 可以写代码了”只是答案的一半

Who cares if pandas syntax is less than perfect, you might say. AI agents can generate the pandas code for us now, so what does it matter? Yes, large language models can save a great deal of time. But generating code is only one part of analytical work. Coding for data analysis is different from software development. It often begins with a question that changes as soon as the first result appears. You filter the data, notice something unexpected, inspect it, revise the grouping, discover missing values, make a chart and then realise that your original question was the wrong one. 你可能会说，谁在乎 pandas 语法是否完美呢？现在 AI 代理可以为我们生成 pandas 代码，那又有什么关系呢？是的，大型语言模型可以节省大量时间。但生成代码只是分析工作的一部分。数据分析的编码与软件开发不同。它通常始于一个问题，而一旦出现第一个结果，问题就会发生变化。你过滤数据，注意到一些意想不到的情况，检查它，修改分组，发现缺失值，制作图表，然后意识到你最初的问题是错误的。

The workflow is not: specification → code → finished product. It is closer to: question → transformation → result → new question → new transformation. That loop is exploratory, creative and interactive. In this setting, there’s value in a human being able to manually transform data with minimal latency. I expect that many analysts are still finding themselves typing small pieces of pandas code into a notebook, even if they are now trusting AI to write larger functions or modules. 工作流程不是：需求说明 → 代码 → 成品。它更接近于：问题 → 转换 → 结果 → 新问题 → 新转换。这个循环是探索性的、创造性的且交互式的。在这种环境下，人类能够以最小的延迟手动转换数据是有价值的。我预计许多分析师仍然会发现自己在笔记本中输入小段的 pandas 代码，即使他们现在信任 AI 来编写更大的函数或模块。

Readability matters

可读性很重要

Secondly, while AI reduces the cost of typing, it doesn’t remove the cost of reading, checking and understanding. A Python data pipeline is also documentation. It tells a colleague—or your future self—what was filtered, which variables were created, and where the final number came from. The easier that path is to follow, the easier it is to review assumptions and catch mistakes. Boilerplate weakens that documentation by lowering the signal-to-noise ratio. The business logic is still present, but it is surrounded by dataframe names, column selectors, quotation marks, lambdas, aliases and API-specific options. 其次，虽然 AI 降低了输入的成本，但它并没有消除阅读、检查和理解的成本。Python 数据流水线也是一种文档。它告诉同事（或未来的你自己）过滤了什么、创建了哪些变量，以及最终数字来自哪里。路径越容易追踪，就越容易审查假设并发现错误。样板代码通过降低信噪比削弱了文档的作用。业务逻辑依然存在，但被 DataFrame 名称、列选择器、引号、lambda、别名和 API 特有的选项所包围。

Let’s be honest: reading other people’s pandas code, especially code that was built through an interactive session can be rather painful. There is no doubt that pandas code is not the most concise or readable way to express the underlying logic of a data pipeline. The rise of AI-generated code only strengthens this argument. If more code is going to be produced automatically, humans need representations that make the generated logic easy to inspect. 老实说：阅读别人的 pandas 代码，尤其是通过交互式会话构建的代码，可能会非常痛苦。毫无疑问，pandas 代码并不是表达数据流水线底层逻辑最简洁或最易读的方式。AI 生成代码的兴起只会加强这一论点。如果更多的代码将自动生成，人类就需要能够使生成的逻辑易于检查的表示方式。

The enduring popularity of visual data tools

可视化数据工具的持久流行

If you’re still not convinced that any of this matters, think for a minute about the popularity of visual data tools. Excel remains embedded in analytical work across almost every industry. Tableau, Power BI, KNIME, Alteryx, Metabase, Orange, RapidMiner and many other products offer different variations on the same promise: touch the data more directly, see feedback quickly and avoid having to translate every thought into a general-purpose programming API. 如果你仍然不相信这一切很重要，请花点时间思考一下可视化数据工具的流行。Excel 仍然嵌入在几乎每个行业的分析工作中。Tableau、Power BI、KNIME、Alteryx、Metabase、Orange、RapidMiner 以及许多其他产品都提供了相同的承诺的不同变体：更直接地接触数据，快速看到反馈，并避免将每一个想法都翻译成通用的编程 API。

I use Excel a lot. There are plenty of occasions when dropping a small dataset into a pivot table is faster than writing a pandas pipeline. Visual tools can be more intuitive and help to reduce the latency from thought to result. Of course, code has a purpose. A script provides an audit trail from raw data to result. It can be reviewed, tested, versioned, rerun and shared. This is why teams move critical work out of spreadsheets in the first place. 我经常使用 Excel。在很多情况下，将小数据集放入数据透视表比编写 pandas 流水线要快得多。可视化工具可以更直观，并有助于减少从想法到结果的延迟。当然，代码有其用途。脚本提供了从原始数据到结果的审计追踪。它可以被审查、测试、版本控制、重新运行和共享。这就是为什么团队最初会将关键工作从电子表格中迁移出来的原因。