Unconstrained CTI-REALM with Open-Weights

Computer Science > Artificial Intelligence arXiv:2610.14180 (cs) [Submitted on 9 Oct 2026] Title: Unconstrained CTI-REALM with Open-Weights Authors: Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma.

计算机科学 > 人工智能 arXiv:2610.14180 (cs) [提交于 2026 年 10 月 9 日] 标题：具有开放权重的无约束 CTI-REALM 作者：Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma。

Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM (arXiv:2603.13517), a benchmark in which an agent turns a threat-intelligence report into a working detection by querying a live telemetry store.

摘要：希望将语言模型代理用于安全工作的组织面临着一种选择：是使用可以自行运行和审计的开放权重模型，还是使用价格昂贵得多的封闭式前沿模型。我们在 CTI-REALM (arXiv:2603.13517) 上研究了这一选择，这是一个基准测试，代理通过查询实时遥测存储库，将威胁情报报告转化为有效的检测结果。

We evaluate twenty-six models, nineteen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns.

我们在统一的测试框架、评分器和大型语言模型评审团下，评估了 26 个模型，其中包括 19 个开放权重模型和 7 个封闭模型。我们的主要发现是，只要给予足够的轮次，开放权重模型可以以不到封闭模型十分之一的成本，达到与最强封闭模型相当的水平。

The benchmark’s published limit on agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished.

该基准测试对代理消息发布的限制是一个混杂因素：耗尽限制的任务在每个检查点上的得分均为零，包括那些已经通过的检查点，因此在该限制下的结果部分衡量了哪些模型能及时完成任务。在已发布的限制下，领先的开放权重模型和封闭模型之间无法区分。

With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to capability: models with similar rewards differ in cost by more than an order of magnitude.

随着限制的放宽，被截断的模型得以恢复，且最高奖励属于开放权重模型。在领先模型中，价格并不能很好地反映能力：奖励相似的模型在成本上可能相差一个数量级以上。

Because the benchmark’s checkpoint scores largely record what an agent did or mentioned, we also introduce stage-level metrics that score what the agent submits, by executing its detection queries in the telemetry store and judging its outputs against the answer key. Rather than combining them into one more overall reward, we show that different models lead different stages, which suggests routing each stage of the workflow to the model that performs it best rather than choosing a single model by its overall score.

由于基准测试的检查点分数主要记录代理做了什么或提到了什么，我们还引入了阶段级指标，通过在遥测存储库中执行代理的检测查询并根据答案键判断其输出来对代理提交的内容进行评分。我们没有将它们合并为一个总奖励，而是展示了不同的模型在不同阶段表现各异，这表明应将工作流的每个阶段路由到表现最好的模型，而不是根据总分选择单一模型。

We also report how many tokens each model generates, and find that the share of output spent on thinking does not track the reward.

我们还报告了每个模型生成的标记数量，并发现用于思考的输出份额与奖励并不相关。