Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

Computer Science > Artificial Intelligence arXiv:2610.14146 (cs) [Submitted on 9 Oct 2026] Title: Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress Authors: Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang.

计算机科学 > 人工智能 arXiv:2610.14146 (cs) [提交于 2026 年 10 月 9 日] 标题：Long-Horizon-Terminal-Bench 2.0：自动化多样化任务构建与衡量持续的智能体进展 作者：Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang。

Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its final state passes a binary check. We present Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), an extension of Long-Horizon-Terminal-Bench that preserves its 46 original tasks while correcting task and evaluation-infrastructure defects and extending 31 tasks. LHTB 2.0 contains 77 tasks across 13 domains and eight task types.

摘要：长程终端基准测试旨在衡量智能体是否能够通过过程奖励在扩展的工作流程中保持进展，而不仅仅是判断其最终状态是否通过二进制检查。我们提出了 Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0)，这是对 Long-Horizon-Terminal-Bench 的扩展，它保留了原有的 46 个任务，同时修正了任务和评估基础设施的缺陷，并扩展了 31 个任务。LHTB 2.0 包含涵盖 13 个领域和 8 种任务类型的 77 个任务。

The extended tasks are produced through a systematic reconstruction-task pipeline. We start from a working scientific or engineering artifact, then use a parameterized reference implementation to generate multiple scenarios, reveal six as behavioral evidence, hold out three for evaluation, and use dense progress ladders to reward partial progress while preventing memorization of public outputs.

扩展任务是通过系统化的重构任务流水线生成的。我们从一个可工作的科学或工程制品出发，利用参数化的参考实现生成多个场景，揭示其中 6 个作为行为证据，保留 3 个用于评估，并使用密集的进度阶梯来奖励部分进展，同时防止对公开输出的记忆。

We also extend the evaluation horizon from 90 minutes to three hours and record verifier checkpoints every 30 minutes, enabling direct measurement of sustained progress within a run. Across 1,145 scored model-task runs from 15 frontier models, GPT-6-astra leads with dense mean reward 0.69 and 37 tasks at R at least 0.95, followed by GPT-5.6-sol with dense mean reward 0.67 and 32 tasks at the same threshold.

我们还将评估周期从 90 分钟延长至 3 小时，并每 30 分钟记录一次验证检查点，从而能够直接衡量单次运行中的持续进展。在来自 15 个前沿模型的 1,145 次评分模型任务运行中，GPT-6-astra 以 0.69 的密集平均奖励和 37 个 R 值至少为 0.95 的任务领先，紧随其后的是 GPT-5.6-sol，其密集平均奖励为 0.67，且有 32 个任务达到相同阈值。

On the 30 reconstruction tasks, binary verifier scores average 0.221, whereas dense ladder scores average 0.653, showing that outcome-only evaluation discards substantial intermediate progress. LHTB 2.0 provides a broader, hardened, and process-aware testbed for studying reliable agent execution over hours.

在 30 个重构任务中，二进制验证器得分平均为 0.221，而密集阶梯得分平均为 0.653，这表明仅基于结果的评估丢弃了大量的中间进展。LHTB 2.0 为研究数小时内的可靠智能体执行提供了一个更广泛、更稳健且具备过程感知能力的测试平台。