AI’s recursive self-improvement might not come so quickly after all
AI’s recursive self-improvement might not come so quickly after all
AI 的递归自我改进或许并不会那么快到来
The AI industry’s boldest promise right now is that AI will soon improve itself, with almost no need for human oversight. LLMs can already write code, generate synthetic data for training, and optimize the computer chips they run on. Forecasts of explosive AI progress predict that what researchers call recursive self-improvement is on the horizon. 目前人工智能行业最大胆的承诺是，AI 将很快实现自我改进，几乎无需人工干预。大语言模型（LLM）已经能够编写代码、生成用于训练的合成数据，并优化其运行所在的计算机芯片。关于 AI 爆发式进步的预测认为，研究人员所称的“递归自我改进”已指日可待。
But a new study suggests that it might take a while for us to get there. The researchers behind it found that AI agents are not yet capable of conducting open-ended AI research—free-form investigations that have no clear-cut answers and require judgment and taste, which may be integral to building self-improving AI. 但一项新的研究表明，我们可能还需要一段时间才能达到这一目标。研究人员发现，AI 智能体目前尚不具备进行开放式 AI 研究的能力——即那些没有明确答案、需要判断力和品味，且对于构建自我改进型 AI 至关重要的自由探索。
A multi-institution group of researchers, led by Peter Kirgis and Sayash Kapoor at Princeton University, found that AI agents could solve the engineering problems necessary to do AI research but lacked the judgment and creativity to produce original research at the caliber of papers accepted by a top machine-learning conference. The gap suggests that some of the hyped-up timelines for automating AI research may be running ahead of the evidence. 由普林斯顿大学的 Peter Kirgis 和 Sayash Kapoor 领导的一个多机构研究小组发现，AI 智能体虽然能够解决 AI 研究中必要的工程问题，但缺乏产生原创研究所需的判断力和创造力，无法达到顶级机器学习会议论文的录用标准。这一差距表明，一些关于 AI 研究自动化的炒作时间表可能超前于实际证据。
Most existing research on how agents can automate AI research evaluates their ability to complete narrow tasks with checkable answers, such as solving engineering problems or post-training small language models against a benchmark. But making progress in AI research also requires open-ended thinking—choosing a set of hypotheses, deciding what evidence would settle a question, or knowing when to start over. 目前大多数关于智能体如何实现 AI 研究自动化的研究，主要评估的是它们完成具有可验证答案的狭窄任务的能力，例如解决工程问题或根据基准测试对小型语言模型进行后训练。但 AI 研究的进展还需要开放式思维——即选择一系列假设、决定什么样的证据能解决问题，或者知道何时该推倒重来。
To test agents on those kinds of skills, the researchers in the study proposed a new method of evaluation called “shadow evaluation,” which requires the AI to answer a research question from a high-quality unpublished paper. The researchers asked Anthropic’s Claude Opus 4.8, running on open-source software called OpenClaw, to tackle such questions, in this case from two papers submitted to the prestigious machine-learning conference NeurIPS 2026. 为了测试智能体在这些方面的能力，研究人员提出了一种名为“影子评估”（shadow evaluation）的新评估方法，要求 AI 回答来自高质量未发表论文中的研究问题。研究人员要求运行在名为 OpenClaw 的开源软件上的 Anthropic Claude Opus 4.8 模型来处理这些问题，这些问题选自提交给著名机器学习会议 NeurIPS 2026 的两篇论文。
The first question was whether a large language model’s “personas,” which determine its behavior, can be controlled by editing the model’s weights (the billions of numbers that store everything it learns during training). The other asked how to design a detector that points out when a model that makes predictions based on spreadsheet data has become unreliable. Because the papers had not been made public, the agents could not memorize the answers from their training data or find them online. 第一个问题是：决定大语言模型行为的“人格”（personas）是否可以通过编辑模型的权重（即存储训练期间所学一切内容的数十亿个数字）来控制。另一个问题是：如何设计一个检测器，指出基于电子表格数据进行预测的模型何时变得不可靠。由于这些论文尚未公开，智能体无法从训练数据中记忆答案，也无法在网上找到它们。
The agents were given six days, $3,000 in Anthropic API credits, a GPU budget to run the experiments, their own virtual computers, and access to the open web to produce a research paper worthy of publication at a top-tier AI conference. The papers’ original authors graded the agents’ papers as they would evaluate one submitted to a conference. Those authors rejected both papers. 智能体获得了六天时间、3000 美元的 Anthropic API 额度、用于运行实验的 GPU 预算、专属虚拟计算机以及访问开放网络的权限，以撰写一篇足以在顶级 AI 会议上发表的研究论文。论文的原始作者像评估会议投稿一样对智能体的论文进行了评分。最终，这些作者拒绝了两篇论文。
The agents were capable of all the engineering required to conduct the research, the human scientists found. The agents reviewed the literature, ran hundreds of experiments, and compiled the results. “On the other hand, the agents were unambiguously bad at carrying out the research itself,” says Kapoor. They ran bizarre experiments (in some cases testing their hypotheses on tiny synthetic datasets), struggled to write intelligibly about their work, and made no novel contribution to their fields. “The papers were nowhere close to the mark when it came to being at the quality of a top AI conference,” he says. 人类科学家发现，智能体能够完成研究所需的所有工程工作。它们查阅了文献、运行了数百次实验并汇总了结果。“但另一方面，智能体在执行研究本身方面表现得非常糟糕，”Kapoor 说。它们进行了奇怪的实验（在某些情况下，在微小的合成数据集上测试假设），难以清晰地描述自己的工作，并且对各自领域没有任何创新贡献。“这些论文在达到顶级 AI 会议质量方面还差得很远，”他说。
That’s because the agents struggled to muster the creativity and judgment necessary for conducting research. They didn’t do enough to explore different ideas, and they committed to unpromising approaches too quickly. Though the agents developed novel and ambitious hypotheses resembling those that the original authors themselves started with, they rejected them on the basis of very limited data. And they couldn’t backtrack from failing approaches. They could make small pivots but could not fundamentally rethink their approach or try new ones from scratch. 这是因为智能体难以调动进行研究所需的创造力和判断力。它们没有充分探索不同的想法，并且过快地投入到没有前景的方法中。尽管智能体提出了与原始作者最初想法相似的新颖且雄心勃勃的假设，但它们却基于非常有限的数据就否定了这些假设。而且它们无法从失败的方法中回溯。它们可以进行小幅调整，但无法从根本上重新思考其方法或从零开始尝试新方法。
The agents also failed to incorporate feedback from subagents or external AI reviewing tools. Instead of revising their methodology, the agents narrowed their claims and added caveats. They also couldn’t effectively use resources, such as tokens, compute, and time. And they couldn’t follow instructions about things like how much time to spend on different phases of the research or how long their paper could be. 智能体也未能整合来自子智能体或外部 AI 审查工具的反馈。它们没有修改方法论，而是缩小了结论范围并增加了注意事项。它们也无法有效利用资源，如 Token、算力和时间。此外，它们无法遵循关于研究不同阶段应花费多少时间或论文长度限制等指令。
For all their failures, the agents didn’t engage in the misbehavior that researchers call “reward hacking,” hiding or misrepresenting experiments or data. Although subagents, or helper AIs that the main agent spawns to handle pieces of the work, occasionally hallucinated or misrepresented the results, these were caught by the orchestrator agent, the lead AI supervising the project. 尽管存在种种失败，但智能体并没有出现研究人员所称的“奖励黑客”（reward hacking）行为，即隐藏或歪曲实验或数据。虽然子智能体（即主智能体为处理部分工作而生成的辅助 AI）偶尔会出现幻觉或歪曲结果，但这些都被编排智能体（负责监督项目的首席 AI）捕捉到了。
The reason AI models are good at research engineering but not at open-ended research may come down to how they’re trained, says Kapoor. Models get good at whatever they can be drilled on in a training regime called reinforcement learning, which is easier to apply to tasks whose success can be checked automatically. “But it’s harder to create environments to train these models when the task itself is open-ended,” he says. Kapoor 表示，AI 模型擅长研究工程却不擅长开放式研究，原因可能归结于它们的训练方式。模型在一种称为强化学习的训练机制下，能够擅长任何可以被反复训练的任务，这种机制更容易应用于可以自动检查成功与否的任务。“但当任务本身是开放式时，就很难创建环境来训练这些模型了，”他说。
Kapoor says the team is now conducting the experiment with Mythos, Anthropic’s most advanced model, which launched in April. It was subsequently required by the Trump administration to meet various safety restrictions and is now available only to approved organizations. Anthropic did not respond to a request for comment. Kapoor 表示，团队目前正在使用 Anthropic 于四月份推出的最先进模型 Mythos 进行实验。该模型随后被特朗普政府要求满足各种安全限制，目前仅向获批组织开放。Anthropic 未回应置评请求。
There are some limitations to the study. It covered just two research papers, and the original authors knew the papers they were grading were generated by AI agents, which could have colored their evaluations. And the researchers had substantial discretion in designing and executing the study, meaning that their preexisting beliefs and biases could have slipped into the results. Evaluations of open-ended research tr 这项研究存在一些局限性。它仅涵盖了两篇研究论文，且原始作者知道他们评分的论文是由 AI 智能体生成的，这可能会影响他们的评估。此外，研究人员在设计和执行研究时拥有很大的自由裁量权，这意味着他们预先存在的信念和偏见可能会渗透到结果中。对开放式研究的评估……