Wrong, not broken / 错误，而非故障

For 20 years, software operations has been built around one question: Is it broken? We’ve gotten very good at answering it. Metrics, logs, and traces. Distributed tracing that can follow a single request through dozens of services. And all of it rests on one sensible assumption: When software fails, the failure eventually shows up as something a machine can measure. 过去 20 年来，软件运维一直围绕着一个核心问题：它坏了吗？我们已经非常擅长回答这个问题。通过指标、日志和追踪，分布式追踪技术甚至可以跟踪单个请求穿过数十个服务的过程。这一切都基于一个合理的假设：当软件发生故障时，这种故障最终会表现为机器可以度量的某种状态。

Now picture an AI agent handling refund requests. It answers in 600 milliseconds. The error rate is zero. And it tells a customer they’re owed $40 when the policy says $140, because the document it retrieved was last quarter’s version. Every dashboard is green. Nothing appears to be broken. It’s wrong. 现在想象一个处理退款请求的 AI 智能体。它在 600 毫秒内做出响应，错误率为零。但它告诉客户应退款 40 美元，而政策规定应为 140 美元，原因在于它检索到的文档是上一季度的版本。此时，所有的仪表盘都是绿色的，没有任何东西看起来“坏了”。但它是错误的。

This week we launched Amazon CloudWatch Omni to observe agents, applications, and infrastructure together, and I wanted to give an inside look at the shift behind it and why it matters. Correctness now has to be measured at the level of the run. 本周，我们发布了 Amazon CloudWatch Omni，旨在将智能体、应用程序和基础设施统一进行观测。我想借此机会深入探讨其背后的转变及其重要性。正确性现在必须在“单次运行”的层面进行衡量。

Software has always been capable of being wrong. What changes with agents is how much of the behavior we care about is resolved while the software is running. An agent adds another set of variables. The same agent, running the same code, can get one request right and the next one wrong. The outcome depends on how the question was phrased, what context was retrieved, which path it took through its tools, and what the model generated. 软件一直都有可能出错。智能体带来的变化在于，我们所关心的行为中有很大一部分是在软件运行过程中才确定的。智能体引入了另一套变量：同一个智能体运行同一段代码，可能对一个请求处理正确，却在下一个请求中出错。结果取决于问题的表述方式、检索到的上下文、它调用工具的路径以及模型生成的内容。

So “is it doing the right thing?” can no longer be answered fully before release. Some of the answer has to come from production, continuously, across the runs themselves. That doesn’t replace the questions observability has always answered. It adds new ones alongside them. Was the work any good? 因此，“它做得对吗？”这个问题无法在发布前得到完全解答。部分答案必须来自生产环境，在每一次运行中持续获取。这并不会取代可观测性一直以来回答的问题，而是在此基础上增加了新的维度：这项工作做得好吗？

The test suite that ran before launch still matters, and now a live measurement can sit alongside it. Did the agent pick the right tool? Did it route the request correctly? When those scores sit on the same trace as latency and token counts, a team can see that a prompt change didn’t slow anything down but did make retrieval answers noticeably worse. 发布前的测试套件依然重要，而现在，实时度量可以与之并存。智能体是否选择了正确的工具？路由请求是否正确？当这些评分与延迟和 Token 计数出现在同一个追踪记录中时，团队就能发现：某次提示词（Prompt）的修改虽然没有降低速度，却明显恶化了检索结果的质量。

This creates a second problem: Quality has to become explicit enough to measure. Human organizations can operate with a surprising amount of tacit judgment. People learn what a good answer sounds like from examples, colleagues, and experience. Not all of it has to be formalized. An evaluator needs something more concrete. What counts as a correct refund decision? When should the agent escalate? Building evaluations therefore forces teams to turn some of that tacit judgment into an operational definition of good. In practice, deciding what should be measured can be as useful as the measurement itself. 这带来了第二个问题：质量必须变得足够明确以便于度量。人类组织可以在很大程度上依赖隐性判断来运作。人们通过案例、同事和经验来学习什么是好的回答，这些并不都需要形式化。但评估器需要更具体的东西：什么样的退款决定才算正确？智能体何时应该升级处理？因此，构建评估体系迫使团队将部分隐性判断转化为“好”的操作定义。在实践中，决定度量什么，往往与度量本身一样重要。

Where in the chain did it go wrong? Go back to the wrong refund. Maybe the model reasoned badly. Or maybe it did everything right, and a payments service three hops downstream was running an old configuration. The team investigating it needs to follow one chain of cause and effect, from the customer’s question, through the agent’s decisions, into the services underneath. Agents are becoming components inside applications, so their behavior needs to be visible alongside the rest of the system. 问题出在链条的哪一环？回到那个错误的退款案例。也许是模型推理失误，又或者是它一切操作正确，但下游三个节点之外的支付服务运行的是旧配置。调查团队需要追踪一条因果链：从客户的问题开始，经过智能体的决策，一直深入到下层的服务。智能体正成为应用程序内部的组件，因此它们的行为必须与系统的其余部分一同被观测。

What’s different about the bad runs? Dashboards remain useful because they make important signals continuously visible. They are especially good when a team already knows what it wants to watch. Agentic systems add important questions that might only become obvious after something unusual happens. Why did refund accuracy drop for European customers this week? Did anything change in retrieval, tool selection, or the services downstream? Being able to ask those questions directly and have the observability system assemble the relevant telemetry changes how an investigation can begin. 异常运行有什么不同？仪表盘依然有用，因为它们能让重要信号持续可见。当团队已经明确知道要监控什么时，它们特别有效。而智能体系统引入了重要的新问题，这些问题往往只有在异常发生后才会显现。为什么本周欧洲客户的退款准确率下降了？检索、工具选择或下游服务是否有变动？能够直接提出这些问题，并让可观测性系统自动汇总相关遥测数据，彻底改变了调查的启动方式。

Did the fix actually work? A bad run in production doesn’t need to end as an incident report. The traces where an agent got something wrong can become a dataset. A team can try a change against those examples, compare the new version with the old one, deploy it, and then see whether production behavior actually improved. That creates a much tighter connection between operating an agent and developing it. The same evidence used to understand a failure becomes part of the test for whether it has been fixed. 修复真的有效吗？生产环境中的一次错误运行不必以事故报告告终。智能体出错的追踪记录可以转化为数据集。团队可以针对这些案例测试修改方案，对比新旧版本，部署后观察生产环境的行为是否真正改善。这在智能体的运维与开发之间建立了更紧密的联系。用于理解故障的证据，同时也成为了验证故障是否已修复的测试的一部分。

More autonomy requires more evidence. Agents get more valuable as they take on more authority. What holds organizations back is whether they can answer a few plain questions. What did the agent do? Was it right? Would we know if that changed? Authority tends to be extended in steps, much like it is with a new colleague. First, every action gets approved. Then only the unusual ones. Then someone reviews a sample. Eventually, the work gets checked after the fact. 更多的自主权需要更多的证据支持。智能体承担的权限越大，价值就越高。阻碍组织发展的往往是能否回答几个简单的问题：智能体做了什么？它做得对吗？如果情况发生变化，我们能察觉吗？权限的授予通常是循序渐进的，就像对待新同事一样：起初，每一个动作都需要审批；然后只审批异常动作；接着由专人抽查；最终，工作在事后进行核查。

Safety works the same way. Permissions still set the outer boundary, and an agent that isn’t allowed to issue refunds won’t. But permissions describe what’s possible, not whether a particular choice was a good one. A policy can say the agent is allowed to issue refunds. It can’t say whether this refund, to this customer, for this amount, was right. More and more of what matters happens inside that boundary. The way to understand it is the same as for quality: Look at what actually happened. 安全性也是如此。权限设定了外部边界，不允许退款的智能体就不会退款。但权限描述的是“什么可行”，而不是“某个选择是否正确”。策略可以规定智能体被允许退款，但无法判断针对某位客户的某笔金额的退款是否合理。越来越多的关键行为发生在边界之内。理解这一点的方法与理解质量的方法相同：观察实际发生了什么。

Put simply, the authority an organization can comfortably give its agents depends on how clearly it can see what they do. That relationship could eventually become dynamic. Today, the decision about how much authority an agent gets is still largely made by people. Looking further ahead, it doesn’t need to stay static. If quality signals are live, autonomy could widen as an agent establishes a track record, and narrow when quality slips, until someone understands why. We are not there yet, but the pieces needed to build systems like this are starting to exist. 简而言之，组织能放心地赋予智能体多大的权限，取决于它能多清晰地洞察智能体的行为。这种关系最终可能会变得动态化。目前，关于智能体权限大小的决策主要还是由人来做。展望未来，这不必保持静态。如果质量信号是实时的，那么当智能体建立起良好的记录时，其自主权可以扩大；而当质量下滑时，自主权则会收窄，直到有人查明原因。我们尚未达到这一步，但构建此类系统所需的组件已经开始出现。

Where we’re investing. This week, we launched Amazon CloudWatch Omni, and it’s our first big step in this direction. It puts agent traces, application services, and infrastructure in one view. It scores agent behavior with 17 built-in evalua… 我们的投入方向：本周，我们发布了 Amazon CloudWatch Omni，这是我们迈向这一方向的第一大步。它将智能体追踪、应用程序服务和基础设施整合在同一个视图中，并利用 17 个内置评估指标对智能体行为进行评分……