Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Long-Horizon-Terminal-Bench：通过基于密集奖励的评分测试智能体在长周期终端任务中的极限
Abstract: AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability.
摘要： 人工智能智能体目前已具备自主完成简短、明确任务的能力。然而，现有的终端基准测试大多集中在几分钟内即可完成的简单问题上，且仅根据最终结果进行评估。这种设置忽略了中间进度和部分解决方案，导致奖励信号稀疏，无法全面反映智能体的能力。
We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows.
我们推出了 Long-Horizon-Terminal-Bench，这是一个包含 46 个长周期任务的终端基准测试，涵盖了实验复现、软件工程、多模态分析、交互式游戏和科学计算等九大类别。每个任务都遵循 Terminal-Bench 风格的设置，配备参考解决方案或模拟引擎，并进一步分解为细粒度的评分子任务。这种设计实现了密集的中间奖励和部分评分，使评估不仅能捕捉智能体是否达到了最终目标，还能衡量其在开放式工作流中的进展程度。
Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks.
Long-Horizon-Terminal-Bench 中的任务通常需要数百个回合以及数分钟到数小时的执行时间，这强调了长周期规划、长上下文管理和迭代调试能力，而非一次性解决问题的能力。我们评估了 15 个前沿模型，发现智能体平均每个任务消耗 990 万个 Token，每次运行约需 231 个回合和 85.3 分钟的执行时间，这使得 Long-Horizon-Terminal-Bench 比以往基于终端的基准测试更具挑战性。
Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
即使是测试中最强的模型，在 0.95 的部分奖励阈值下也仅达到 15.2% 的 pass@1，在 1.0 的完美奖励阈值下仅达到 10.9%；而所有模型的平均通过率在这两个阈值下分别为 4.3% 和 1.7%。这些结果表明仍有很大的提升空间。我们进一步分析了故障模式和错误类型，并发布了 Long-Horizon-Terminal-Bench，以支持长周期终端智能体的未来发展。