Your AI Agent Passed Every Eval. Finance Still Killed It.
Your AI Agent Passed Every Eval. Finance Still Killed It.
你的 AI Agent 通过了所有评估，但财务部门依然砍掉了它。
Agentic AI. I published a twelve-metric eval harness for AI agents two months ago. It measures whether the agent is right. It does not measure whether the agent pays for itself, and that gap is where accurate agents die.
智能体 AI。两个月前，我发布了一套针对 AI 智能体的十二项评估指标框架。它衡量的是智能体是否“正确”，但它无法衡量智能体是否“物有所值”。正是这一差距，成为了那些表现精准的智能体被淘汰的坟墓。
An agent can pass every quality metric and still cost more than it is worth. The agent passed every metric in the eval harness. Then it got shut down. I was in the room for the review that killed it. A mid-market SaaS company had built a customer-operations agent to handle billing and account tickets. They had instrumented it against the twelve-metric evaluation framework I published on this site two months ago. Task completion, faithfulness, retrieval precision, tool-call accuracy, latency, hallucination rate. Every metric was inside its target band. The agent was, by every measure the harness tracked, working.
一个智能体可能通过了所有质量指标，但其成本依然高于其价值。该智能体通过了评估框架中的每一项指标，但随后还是被关停了。我当时就在那场决定其命运的评审会议现场。一家中型 SaaS 公司开发了一个客户运营智能体，用于处理账单和账户工单。他们按照我两个月前发布在该网站上的十二项评估框架对其进行了测试。任务完成度、忠实度、检索精度、工具调用准确性、延迟、幻觉率——每一项指标都在目标范围内。按照该框架追踪的所有标准来看，这个智能体是“正常工作”的。
The CFO opened her laptop and put one number on the screen. The cost per resolved ticket, including the tickets the agent tried and failed to resolve, was higher than what the company had been paying humans to do the same work. Not by a rounding error. The agent-assisted workflow was costing the company more per resolved ticket than the fully-human workflow it was supposed to replace.
首席财务官（CFO）打开笔记本电脑，在屏幕上展示了一个数字。每个已解决工单的成本（包括智能体尝试但未能解决的工单）高于公司此前支付给人工处理同样工作的费用。这绝非四舍五入的误差。智能体辅助的工作流在处理每个工单上的成本，比它本应取代的全人工工作流还要高。
The head of engineering pushed back. Every quality metric is green, he said. The agent is more accurate than the team was on the same ticket types. That was true. It was also, in that room, irrelevant. The agent was accurate and economically underwater, and the eval harness had no metric that could see the second half of that sentence.
工程负责人试图反驳。他说，每一项质量指标都是绿色的，智能体在处理同类工单时比团队更准确。这确实是事实，但在那个房间里，这毫无意义。智能体虽然精准，但在经济上却入不敷出，而评估框架中没有任何指标能反映出后半部分的问题。
I built that harness. I stand by the twelve metrics. But I got something wrong, and two months of watching teams deploy against it has made the gap impossible to ignore. The harness measures whether the agent is right. It does not measure whether the agent pays for itself. In 2026, that second question is the one that decides which agents stay in production, and it is not in the framework.
这套框架是我构建的，我依然支持这十二项指标。但我忽略了一些东西，过去两个月观察各团队的部署情况，让我无法再忽视这个差距。该框架衡量的是智能体是否“正确”，却没衡量它是否“回本”。在 2026 年，后一个问题才是决定哪些智能体能留在生产环境的关键，而它并不在我的框架内。
What the harness measures, and the half it misses
框架衡量了什么，又遗漏了什么
The twelve-metric harness was built to answer a specific question: is this agent behaving correctly in production? It covers the failure modes that make an agent wrong. Retrieval that surfaces the wrong context. Generation that hallucinates. Tool calls that construct bad arguments. Reasoning loops that never terminate. Outputs that drift from the user’s intent. Each metric isolates a way the agent can produce a wrong result, and together they give you a defensible read on quality.
这套十二项指标框架旨在回答一个特定问题：该智能体在生产环境中的表现是否正确？它涵盖了导致智能体出错的各种故障模式：检索到错误的上下文、生成幻觉内容、工具调用参数错误、推理循环无法终止、输出偏离用户意图等。每一项指标都隔离出一种可能导致错误结果的方式，综合起来，它们能为你提供一份可信的质量评估报告。
Every one of those metrics is a quality metric. Not one of them is an economic metric. This did not feel like a gap when I wrote the framework. Quality was the hard problem. Cost felt like an operational detail you could read off a billing dashboard whenever you wanted it. Token costs were falling every quarter. The narrative in early 2026 was that inference was getting cheaper and the real challenge was reliability. Measure quality, the thinking went, and the economics will take care of themselves.
这些指标全部属于质量指标，没有一个是经济指标。当我编写该框架时，并没有觉得这是一个缺失。当时，“质量”是难题，而“成本”感觉只是随时可以从账单仪表盘上读取的操作细节。Token 成本每个季度都在下降。2026 年初的主流观点是：推理成本正在降低，真正的挑战在于可靠性。当时的逻辑是：只要衡量好质量，经济效益自然会水到渠成。
The economics did not take care of themselves. Inference costs per token did keep falling, but agent architectures got hungrier faster than tokens got cheaper. A single agent resolution in 2026 is not one model call. It is a plan step, several tool calls, a few reasoning passes, sometimes a self-critique loop, sometimes a retry after a failed tool call, and a final synthesis. The token price went down and the tokens per resolution went up, and for a lot of workflows the second effect won.
但经济效益并没有水到渠成。虽然每个 Token 的推理成本确实在持续下降，但智能体架构对 Token 的消耗速度远超成本下降的速度。在 2026 年，一次智能体解决问题的过程不再仅仅是一次模型调用，它包含规划步骤、多次工具调用、几轮推理过程、有时还有自我反思循环、工具调用失败后的重试，以及最终的综合输出。Token 价格下降了，但单次解决问题所需的 Token 数量却上升了，对于许多工作流来说，后者的影响占据了主导。
That is the structural reason a quality-only harness misleads. It tells you the agent is good at the job. It says nothing about whether the job, done this way, is worth doing. An agent can sit comfortably inside every quality band in the framework and still lose money on every unit of work it completes. The harness will show you all green while the unit economics bleed, because the harness was never pointed at the unit economics.
这就是仅靠质量评估框架会产生误导的结构性原因。它告诉你智能体工作做得很好，但它没告诉你用这种方式完成这项工作是否值得。一个智能体可以完美符合框架内的所有质量标准，却在完成每一项工作时都在亏钱。当单位经济效益在流血时，框架却显示全绿，因为该框架从未指向单位经济效益。
The metric: cost per successful outcome
关键指标：单次成功产出的成本
The metric the harness needed is cost per successful outcome. Not cost per call. Not cost per token. Not accuracy. The fully-loaded cost of producing one successful business outcome, divided by the number of successful outcomes. The word that does the work in that definition is successful. Cost per call is easy and misleading. Cost per successful outcome is hard and honest, and the difference between them is where the SaaS company’s economics went wrong.
该框架所缺失的指标是“单次成功产出的成本”。不是单次调用成本，不是单次 Token 成本，也不是准确率。它是产生一个成功业务结果的完全分摊成本，除以成功结果的数量。这个定义中起关键作用的词是“成功”。单次调用成本很容易计算，但具有误导性；单次成功产出成本计算困难，但却真实。两者之间的差异，正是那家 SaaS 公司经济账算错的地方。
Here is the arithmetic that killed their agent, with the numbers rounded to protect the client but the structure intact. The agent attempted every incoming ticket. Each attempt cost about $3.40 in fully-loaded inference: the reasoning passes, the tool calls to their billing system, the retries, the synthesis. The agent resolved 71 percent of tickets to a quality bar the eval harness confirmed was high. The other 29 percent, it escalated to a human after attempting them.
以下是导致其智能体被砍掉的算术逻辑（为保护客户隐私，数字已取整，但结构保持不变）。智能体尝试处理每一个收到的工单。每次尝试的完全分摊推理成本约为 3.40 美元：包括推理过程、对账单系统的工具调用、重试以及最终综合。智能体解决了 71% 的工单，且质量达到了评估框架确认的高标准。其余 29% 的工单在尝试后被转交给了人工。
Take a thousand tickets. The agent spends $3,400 attempting all of them. It resolves 710 and escalates 290. Those 290 escalations then go to a human agent at roughly $4.20 of loaded cost each, another $1,218. The company paid $4,618 to resolve a thousand tickets that would have cost $4,200 with humans alone. The agent-assisted workflow was 10 percent more expensive than the workflow it replaced.
以一千个工单为例。智能体花费 3,400 美元尝试处理所有工单。它解决了 710 个，转交了 290 个。这 290 个转交工单由人工处理，每单完全分摊成本约为 4.20 美元，共计 1,218 美元。公司总共支付了 4,618 美元来解决这一千个工单，而如果完全由人工处理，成本仅为 4,200 美元。智能体辅助的工作流比它所取代的工作流贵了 10%。
Now compute the metric that matters. Cost per successful agent outcome is not the $3.40 per attempt that shows up on the inference dashboard. It is the total agent spend divided by the successful resolutions: $3,400 over 710, which is $4.79 per resolved ticket. Every ticket the agent successfully resolved effectively cost $4.79, because the cost of the attempts that failed and escalated has to be carried by the attempts that succeeded.
现在计算那个真正重要的指标。单次成功智能体产出的成本，并不是推理仪表盘上显示的每次尝试 3.40 美元。它是智能体的总支出除以成功解决的数量：3,400 美元除以 710，即每个已解决工单 4.79 美元。智能体成功解决的每一个工单，实际成本为 4.79 美元，因为那些失败并转交的尝试成本，必须由成功的尝试来分摊。