The gate for an agent belongs in the environment, not in the agent
The gate for an agent belongs in the environment, not in the agent
智能体的“闸门”应设在环境之中,而非智能体本身
There’s a discussion running on Product Hunt right now about where an AI agent should stop and hand a decision back to a person. The framing that stuck with me: “can this be undone?” is the wrong gate. It over-fires on things nobody cares about, like writing a log line, and under-fires on the ones that actually hurt. A mass email. A migration you can only roll back with downtime. Product Hunt 上目前正在讨论一个话题:AI 智能体在什么情况下应该停止操作并将决策权交还给人类?其中一个让我印象深刻的观点是:“这可以撤销吗?”是一个错误的判断标准。它会导致在无关紧要的事情(比如写一行日志)上过度触发,而在真正造成伤害的事情(比如群发邮件,或者需要停机才能回滚的数据库迁移)上却触发不足。
The proposed replacement is reach. Not “is this safe” but “how far does this go if I’m wrong.” And the sharpest point in that thread is that the agent is the worst possible judge of its own reach, because it doesn’t know there are 50,000 people on the list. 建议的替代标准是“影响范围”(reach)。不是问“这安全吗”,而是问“如果我错了,这会造成多大的影响”。该讨论中最深刻的一点是:智能体是评估自身影响范围的最差人选,因为它根本不知道邮件列表里有 50,000 个人。
I’ve been building a browser with an agent in it, and the browser case makes that concrete in a way the terminal case doesn’t. A coding agent’s blast radius is usually one person: you, reading a diff. A browser agent doesn’t start there. It inherits every session you’re already signed into. The reach of a click isn’t a property of the agent’s plan, it’s a property of the cookie jar it’s holding. 我一直在开发一款内置智能体的浏览器,浏览器场景让这个问题变得比终端场景更加具体。编程智能体的“爆炸半径”通常只有一个人:即正在查看代码差异(diff)的你。但浏览器智能体并非如此,它继承了你已经登录的所有会话。点击操作的影响范围并非取决于智能体的计划,而是取决于它所持有的 Cookie 容器。
The same “click the blue button” step is harmless on a docs site and irreversible on a payments dashboard, and nothing in the agent’s own summary distinguishes them. So I stopped trying to make the agent classify risk. Here’s what I do instead, with the caveat that this is alpha and I’ve been wrong about it before. 同样的“点击蓝色按钮”操作,在文档网站上是无害的,但在支付后台却是不可逆的,而智能体自身的总结中没有任何东西能区分这两者。所以我不再试图让智能体去进行风险分类。取而代之,我采取了以下做法——需要说明的是,这还处于 Alpha 阶段,我以前也曾在这方面犯过错。
Some things never reach the model. Credential fields are filtered out of the page snapshot before the agent sees it. That means password inputs, and anything with cc-number, cc-csc, cc-exp, or one-time-code autocomplete semantics. A fill targeting a password field refuses outright. The agent cannot misjudge the reach of a control it was never shown, and that’s a stronger guarantee than any classifier, because it doesn’t depend on getting a judgment right. 有些信息根本不会传给模型。在智能体看到页面快照之前,凭证字段会被过滤掉。这意味着密码输入框,以及任何带有信用卡号、安全码、过期日期或一次性验证码自动填充语义的字段都会被屏蔽。如果尝试填充密码字段,系统会直接拒绝。智能体无法误判它从未见过的控件的影响范围,这比任何分类器都更可靠,因为它不依赖于模型是否做出了正确的判断。
The model’s judgment is a floor, never a ceiling. For everything that does reach it, the agent can say “this needs approval” and be believed. It cannot say “this is routine” and be believed. A structurally sensitive control still escalates to a native confirmation even when the model called the action ordinary. The agent can raise the gate; it can’t lower it. 模型的判断是底线,绝非上限。对于传给模型的信息,如果智能体说“这需要批准”,它是可信的;但如果它说“这是常规操作”,则不可信。即使模型认为某个操作是普通的,如果该控件在结构上属于敏感操作,系统依然会升级为原生确认。智能体可以提高闸门,但不能降低它。
Sensitivity is read off the DOM, not off the plan. What marks a control sensitive is its type and autocomplete semantics, which is evidence in the page itself, rather than the agent’s description of what it means to do. A timeout is a denial, not a wait. A stalled job is a silent failure; a denied one is loud, and loud failures get fixed. 敏感性是从 DOM 中读取的,而不是从计划中读取的。标记一个控件是否敏感的依据是它的类型和自动填充语义,这些是页面本身提供的证据,而不是智能体对自己意图的描述。超时意味着拒绝,而不是等待。停滞的任务是静默失败;而拒绝则是显性的,显性的失败更容易被修复。
The part I haven’t solved is denied-and-resumable. Discarding the work is clean and useless, because people want to come back twenty minutes later and continue. But that means storing the world as it was when the question was asked, and the world moved. In a browser that’s especially unkind: the DOM you asked about doesn’t exist anymore. I don’t have a good answer for it yet. 我尚未解决的部分是“被拒绝后可恢复”的问题。直接丢弃工作虽然干净利落但毫无用处,因为用户希望二十分钟后回来继续操作。但这需要存储提问时页面的状态,而世界是在不断变化的。在浏览器中,这尤其棘手:你之前询问的 DOM 可能已经不存在了。对此,我还没有好的解决方案。