Handbook.md shows that long policy documents do not reliably govern agents
Handbook.md shows that long policy documents do not reliably govern agents
Handbook.md 研究表明：长篇政策文档无法可靠地约束 AI Agent
Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon.
语言模型 Agent 正越来越多地在“常设指令”下部署：即通过将系统提示词、政策文件或技能文档置于上下文中，并信任 Agent 会让这些内容约束其后续的所有行为。现有的基准测试很少直接测试这种部署模式；它们通常只衡量 Agent 是否能完成任务，而不是衡量一份冗长且具有约束力的政策文档是否真的能在长期的工具使用过程中限制其行为。
We present this http URL, a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages.
我们提出了 HANDBOOK.md，这是一个包含 65 个 Agent 任务的基准测试，其模型参考了企业员工遵循公司手册的方式。每个任务都将 Agent 置于一个独立的虚拟公司环境中，包括文件工作区以及通过模型上下文协议（Model Context Protocol）暴露的模拟电子邮件、聊天、日历、问题追踪和商业服务，并要求其在由专家编写的 20 到 124 页标准作业程序（SOP）的指导下执行日常专业工作。
Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not.
这些任务涵盖了五个领域（金融、医疗账单、保险、物流和人力资源）以及十家虚构公司。为了防止模型通过记忆作弊，每个任务都会修改十本基础手册中的一本，改变评分所依据的具体规则和阈值，因此没有任何两个任务共享同一套政策。评分过程是完全确定性的：每个任务都有一套程序化标准（共 824 项），用于检查必要的操作是否已执行，以及禁止的操作是否未发生。
Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve.
在严格的评分标准下（即只有满足所有标准才算通过），在评估的 30 种模型配置中，表现最好的仅通过了 36.2% 的测试，而大多数前沿模型配置的通过率甚至不到 25%。失败案例呈现出一致的模式：Agent 往往会因为环境中看似合理的请求而忽略常设政策、在执行必要检查后却违背检查结果行事、在长周期任务中遗忘规则细节，以及谎报其并未达成的合规性。