Nvidia just showed that the harness, not the AI model, is now the real hero
Nvidia just showed that the harness, not the AI model, is now the real hero
英伟达最新研究表明：AI 的“外挂框架”才是真正的英雄，而非模型本身
Nvidia published some interesting new research on Friday suggesting it’s the harness, more than the underlying model, that is far more important when asking an AI to do long-horizon tasks. A harness is the software wrapper around an AI model — the tools, memory management, and rules that turn a raw model into something that can act on its own. 英伟达周五发布了一项有趣的新研究，表明在要求 AI 执行长周期任务时，“外挂框架”（harness）比底层模型重要得多。所谓“外挂框架”，是指围绕 AI 模型构建的软件包装层——包括工具、内存管理和规则，它们将原始模型转化为能够自主行动的系统。
The TL;DR: Simply by using a custom harness tweaked to handle memory well and including a “supervisor” boss-like component, researchers got Claude Opus 5 to achieve a 100% score on the interactive reasoning benchmark ARC-AGI-3 — a set of 2D games with no instructions, where the model has to figure out how to play and win, similar to how a human would. (That’s a benchmark that has particularly irked rival frontier lab OpenAI.) Without the harness, Opus 5 scored 30%, which was the top result among all the models tested. 简而言之：研究人员通过使用一个经过优化以处理内存的自定义框架，并加入了一个类似“主管”的组件，使 Claude Opus 5 在交互式推理基准测试 ARC-AGI-3 上取得了 100% 的分数。ARC-AGI-3 是一组没有任何说明的 2D 游戏，模型必须像人类一样摸索出玩法并获胜。（这一基准测试曾让竞争对手 OpenAI 深感头疼。）在没有该框架的情况下，Opus 5 的得分仅为 30%，但这已是所有测试模型中的最高分。
Nvidia’s research is another indicator that, while model choice does matter, the model itself — the part that acts as the agent’s “brain” — is a smaller part of an agentic system than many AI users realize, especially for long-horizon tasks. The harness is what makes a model an agent: It handles memory, context, and feedback. 英伟达的研究再次表明，虽然模型选择确实重要，但作为智能体“大脑”的模型本身，在智能体系统中所占的比重远比许多 AI 用户意识到的要小，尤其是在处理长周期任务时。正是外挂框架让模型成为了智能体：它负责处理内存、上下文和反馈。
“Generally speaking, the world interprets an agent almost as an API of the model,” Adel El Hallack, vice president of product in Nvidia’s AI unit (pictured above), tells TechCrunch. But an agent is actually more than that. “It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to.” “总的来说，外界几乎将智能体解读为模型的 API，”英伟达 AI 部门产品副总裁 Adel El Hallack（上图）告诉 TechCrunch。但智能体实际上远不止于此。“它不仅是模型，还是模型周围的脚手架，也就是我们所说的外挂框架，即它所利用的一系列工具。它还包括运行时环境以及我们赋予它访问权限的相关技能和库。”
Long-horizon tasks are those that require stringing many decisions together, sometimes over days, to produce completed work. This is in contrast to an AI just spitting out a response to a prompt. Figuring out how to get an AI to do long-horizon tasks without getting distracted and going off in la-la land is one of the holy grails in agentic research. 长周期任务是指需要将许多决策串联起来，有时甚至需要数天时间才能完成的工作。这与 AI 仅仅针对提示词输出回复截然不同。如何让 AI 执行长周期任务而不至于“走神”或陷入混乱，是智能体研究中的“圣杯”之一。
For example: Microsoft published research in April that tested 19 LLMs on long-horizon tasks involving document editing and discovered that all the models, including frontier ones, filled the documents with errors. (If humans produced work like that, they would be promptly fired.) Models stringing decisions together on their own have also been caught deleting their users’ files, even whole databases or turning to criminal behavior to achieve their objectives from collusion to hacking. 例如：微软在四月份发布的一项研究测试了 19 个大语言模型在文档编辑等长周期任务上的表现，结果发现所有模型（包括前沿模型）都在文档中充满了错误。（如果人类产出这样的工作，早就被解雇了。）自主串联决策的模型还曾被发现会删除用户文件，甚至整个数据库，或者为了实现目标而采取从串通到黑客攻击等犯罪行为。
The choice by Nvidia researchers to use this interactive reasoning benchmark for their tests is particularly meaningful, almost funny. A 100% score means that the model can beat the games as well as humans. OpenAI was so flustered by its models’ abysmal scores (less than 10%) on ARC-AGI-3 that it conducted its own research last month. Like Nvidia, OpenAI discovered that simply by tweaking two settings on the harness, its models tripled their scores. But none of the models came close to hitting a 100% score, like Nvidia’s researchers achieved. 英伟达研究人员选择使用这一交互式推理基准进行测试，显得格外有意义，甚至有些讽刺。100% 的分数意味着模型在游戏中的表现与人类相当。OpenAI 对其模型在 ARC-AGI-3 上惨淡的得分（低于 10%）感到非常不安，以至于在上个月进行了自己的研究。与英伟达一样，OpenAI 发现只需调整框架中的两个设置，其模型的得分就能翻三倍。但没有一个模型能像英伟达研究人员那样达到 100% 的分数。
They showed that the harness needs a “supervisor” component that prods the agent in the right direction if it gets stuck. “The more interesting part was introducing a supervising agent in addition to your main agent that’s doing the work,” El Hallack said. It “almost acts like a CEO to nudge the agent when it goes off direction or starts exploring a path that it might lead to a dead end, or re-explore a path that it had previously trod.” 他们证明了外挂框架需要一个“主管”组件，在智能体卡住时引导其走向正确的方向。“更有趣的部分是在执行任务的主智能体之外，引入了一个监督智能体，”El Hallack 说。它“几乎就像一位 CEO，当智能体偏离方向、开始探索可能导致死胡同的路径，或者重复探索之前走过的路径时，它会对其进行提醒。”
While the concept of the supervising agent isn’t exactly new, today most agent users are relying on only one layer for their harness, like Claude Code, Codex, or Hermes. Nvidia researchers created their own souped-up harness called the Agentic Variation Operators (AVO). Note that this isn’t a new Nvidia product. Nvidia instead produces lots of open bits and pieces of tech for building harnesses under the Nemo brand. Some of that tech is commercial, much is openly available. 虽然监督智能体的概念并不新鲜，但目前大多数智能体用户在构建框架时仅依赖单层结构，例如 Claude Code、Codex 或 Hermes。英伟达研究人员创建了他们自己的增强型框架，称为“智能体变异算子”（AVO）。请注意，这不是英伟达的新产品。相反，英伟达在 Nemo 品牌下生产了许多用于构建框架的开源技术组件。其中一些技术是商业化的，但大部分是公开可用的。
Still, Nvidia’s results add to the growing evidence that model choice is far from the only factor in agentic performance. In July, for instance, Databricks published some stunning research that shows that the harness, more than model, dramatically impacts AI costs. “You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness,” Databricks CEO Ali Ghodsi told TechCrunch. “So you think, oh, this is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your cost.” 尽管如此，英伟达的研究结果进一步证明，模型选择远非影响智能体性能的唯一因素。例如，Databricks 在七月份发布了一项惊人的研究，表明外挂框架对 AI 成本的影响远大于模型本身。“你可以选择同一个模型，但使用不同的框架，如果框架选错，成本会显著增加，”Databricks 首席执行官 Ali Ghodsi 告诉 TechCrunch。“所以你可能会想，哦，这是一个昂贵的模型，这是一个便宜的模型。但等等，你用的是什么框架？这本身就能让你的成本翻倍。”
Nvidia’s larger point is to show that open harnesses, like open models, put users in control far more than they realize. “We believe, and we’re demonstrating with the ecosystem, how open harnesses allow you to turn a lot more knobs to drive up that accuracy,” El Hallack said. “It relates to OpenAI slowing down the training of their models,” as a result of models creating security breaches. “We believe in having an open agent stack — where you have control across the harness, across the infrastructure, across the runtime — is what’s required for us to usher the ecosystem forward and securely,” he added. 英伟达更深层的意图是展示开源框架（就像开源模型一样）能让用户获得比预期更多的控制权。“我们相信，并且我们正在通过生态系统证明，开源框架如何让你能够调整更多的参数来提高准确性，”El Hallack 说。“这与 OpenAI 放慢模型训练速度有关，”因为模型会导致安全漏洞。“我们相信拥有一个开放的智能体技术栈——即你在框架、基础设施和运行时环境上都拥有控制权——是我们推动生态系统安全向前发展的必要条件，”他补充道。