Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Harbor Adapters 和 Harbor-Index：用于大规模智能体评估的基础设施与精选元数据集
Abstract / 摘要
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions.
在日益增多的智能体基准测试中评估智能体是一项挑战，因为它们通常需要复杂的环境和智能体集成。我们推出了 Harbor Adapters，这是一个用于智能体基准测试的统一评估基础设施。我们的工作做出了三项贡献。
First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible.
首先，我们开发了基准适配器（benchmark adapters），将 80 多个基准测试移植到可评估任意智能体的平台上，并通过严格的代码审查和对等实验进行了验证。其次，我们对跨越不同能力层级的 8 个模型在 54 个基准测试中进行了大规模评估；每个模型均使用 Terminus-2 和 3 种原生工具之一运行。这使得我们能够比以往更广泛地分析智能体的能力和故障模式。
Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%.
第三，我们推出了 Harbor-Index，这是一套精选的包含 82 个困难、多样且高质量任务的集合，涵盖了 29 个基准测试。这些任务是通过难度过滤、人工智能与人工审核以及“审核-修复”循环从适配后的套件中提炼出来的。Harbor-Index 在保持大规模智能体评估的挑战性和广度的同时，运行成本也更加可控；在所有评估的模型与工具配置中，没有一个通过率超过 30%，最强的配置（GPT-5.5 配合 Codex）达到了 28.0%。
We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.
我们将适配器、评估结果、深度分析和 Harbor-Index 作为开源项目发布，以支持对语言模型智能体进行更可靠、更全面的评估。