RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
Computer Science > Computation and Language arXiv:2610.22507 (cs) [Submitted on 15 Oct 2026] Title:RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents Authors:Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du, Weicheng Gu, Fanglin Xu, Yi Xing, XiangLong Liu, Jujie He, Zhange Zhang, Huatong Song, Zhaoyi Li, Deyang Kong, Chuan Hao, Peihao Wu, Mingjie Tang, Ran Tao, Jian Yang, Wayne Xin Zhao, Bryan Dai.
计算机科学 > 计算与语言 arXiv:2610.22507 (cs) [提交于 2026 年 10 月 15 日] 标题：RealCLI：迈向 CLI 智能体人机协作的统一评估。作者：Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du, Weicheng Gu, Fanglin Xu, Yi Xing, XiangLong Liu, Jujie He, Zhange Zhang, Huatong Song, Zhaoyi Li, Deyang Kong, Chuan Hao, Peihao Wu, Mingjie Tang, Ran Tao, Jian Yang, Wayne Xin Zhao, Bryan Dai。
Abstract: Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks.
摘要：命令行界面 (CLI) 智能体越来越多地执行长周期任务，然而现有的基准测试要么假设用户指令在执行过程中保持不变，要么在评估任务结果时未能捕捉运行时的协作质量。在本文中，我们提出了 REALCLI，这是一个配对基准测试，旨在统一评估现实 CLI 任务中的任务执行情况与人机协作质量。
To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions.
为了全面衡量协作，REALCLI 构建了一个五维框架，并将其细化为 22 个子维度和 51 个交互场景模板。REALCLI 提供了两种配对比较模式：REALCLI-EXEC 在开始时提供完整的规范，而 REALCLI-COLLAB 则引入了一个脚本化用户，用于提供渐进式信息、修正、约束修订和授权决策。
Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score ($J_{\mathrm{collab}}$), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management.
评估揭示了前沿智能体在能力上的巨大差距：最佳配置的平均得分仅为 49.2%，而当使用联合协作分数 ($J_{\mathrm{collab}}$) 进行评估时，峰值性能仅达到 37.6%。这些结果表明，任务有效性与协作质量之间存在显著差异，模型在可控性、信任校准和安全边界管理方面始终表现欠佳。
REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents. Subjects: Computation and Language (cs.CL).
REALCLI 为衡量和推进协作式 CLI 智能体提供了一个严谨且可复现的基础。学科：计算与语言 (cs.CL)。