AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks
Computer Science > Artificial Intelligence arXiv:2610.20675 (cs) [Submitted on 14 Oct 2026] Title: AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks Authors: Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu, Yi Xie, Nico Pelleriti, Shiyang Liu, Menghan Lu, Max Zimmer, Brando Miranda, Jiangchao Yao, Bo Liu, Sanmi Koyejo, Sebastian Pokutta, Bo Han.
计算机科学 > 人工智能 arXiv:2610.20675 (cs) [提交于 2026 年 10 月 14 日] 标题：AlphaDiana：可验证推理任务中智能体的测量系统 作者：Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu, Yi Xie, Nico Pelleriti, Shiyang Liu, Menghan Lu, Max Zimmer, Brando Miranda, Jiangchao Yao, Bo Liu, Sanmi Koyejo, Sebastian Pokutta, Bo Han。
Abstract: Agents pair reasoning models with harnesses that organize model calls, execute tools, load skill instructions, and retain history in memory. Understanding agent performance requires linking execution conditions to outcomes, behavior, and resource use. We introduce AlphaDiana, a measurement system with modular interfaces for datasets, agents, environments, and scorers. It preserves harness-native control loops while controlling file reuse and memory retention separately.
摘要：智能体将推理模型与用于组织模型调用、执行工具、加载技能指令以及在内存中保留历史记录的工具框架（harnesses）相结合。了解智能体的性能需要将执行条件与结果、行为和资源使用情况联系起来。我们引入了 AlphaDiana，这是一个为数据集、智能体、环境和评分器提供模块化接口的测量系统。它在保留工具框架原生控制循环的同时，分别控制文件重用和内存保留。
AlphaDiana supports macro analyses of complete agents and micro interventions on agent components such as tools, skills, and memory under matched conditions. We study three open-source models and three open-source harnesses on five answer-based benchmarks, with two models also tested on two interactive benchmarks. We find that harnesses do not consistently improve the accuracy of reasoning: gains on questions unsolved by the model alone can be outweighed by losses on questions it answers correctly without tools.
AlphaDiana 支持在匹配条件下对完整智能体进行宏观分析，以及对工具、技能和内存等智能体组件进行微观干预。我们在五个基于答案的基准测试中研究了三个开源模型和三个开源工具框架，其中两个模型还在两个交互式基准测试中进行了测试。我们发现，工具框架并不能持续提高推理的准确性：在模型单独无法解决的问题上获得的收益，可能会被其在没有工具辅助下本能正确回答的问题上的损失所抵消。
This inconsistency extends to component interventions: neither skill guidance nor retained history reliably improves accuracy. Behavior and output-token use must be assessed with correctness: low token entropy can accompany wrong answers, and less output per execution can mean more output tokens per correct answer when accuracy falls. AlphaDiana links answer correctness and execution status to traces and resource records for comparing agents and diagnosing failures under shared task and resource policies.
这种不一致性也延伸到了组件干预上：技能指导和保留的历史记录都无法可靠地提高准确性。行为和输出标记（token）的使用必须结合正确性进行评估：低标记熵可能伴随着错误的答案，而当准确率下降时，每次执行较少的输出可能意味着每个正确答案需要更多的输出标记。AlphaDiana 将答案正确性和执行状态与追踪记录及资源记录相关联，以便在共享任务和资源策略下比较智能体并诊断故障。