Measuring Cross-Task Behavioral Consistency in Language Model Agents

衡量语言模型智能体跨任务的行为一致性

Abstract: Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it behaves. We argue that behavioral consistency across tasks is a distinct and measurable property, and we introduce the Behavioral Consistency Metric (BCM) to quantify it.

摘要： 智能体评估几乎完全依赖于成功率等结果指标，这些指标只能捕捉智能体是否成功，却无法衡量其行为的一致性。我们认为，跨任务的行为一致性是一个独特且可衡量的属性，并引入了行为一致性指标（Behavioral Consistency Metric, BCM）来对其进行量化。

BCM trains a model to predict task success from behavioral features of agent execution traces, derives a per-trajectory feature-attribution vector, and measures the mean pairwise similarity of these vectors within an agent system.

BCM 通过智能体执行轨迹的行为特征训练模型来预测任务成功率，推导出每个轨迹的特征归因向量，并测量智能体系统中这些向量的平均成对相似度。

Across roughly 9,000 trajectories from six language model agents on software engineering tasks, our central finding is that cross-task and within-task consistency are distinct axes that can diverge: some systems are locally reproducible, behaving similarly on repeated attempts at one task, yet globally fragmented, with no stable strategy across different tasks, while others are consistent at both scales.

通过对六个语言模型智能体在软件工程任务中的约 9,000 条轨迹进行分析，我们的核心发现是：跨任务一致性和任务内一致性是两个可以分化的独立维度。一些系统在局部具有可重复性，即在重复尝试同一任务时表现相似，但在全局上却是碎片化的，在不同任务间缺乏稳定的策略；而另一些系统则在两个尺度上都保持一致。

Prior work measures only same-task reproducibility and so cannot observe this separation. We further find that consistency is not reducible to success rate, since systems with comparable success can differ sharply in consistency, and that the frontier-versus-open-source consistency gap persists under a within-task control that holds task difficulty constant.

以往的研究仅衡量同一任务内的可重复性，因此无法观察到这种差异。我们进一步发现，一致性不能简单地归结为成功率，因为成功率相当的系统在一致性上可能存在巨大差异；此外，在控制任务难度不变的情况下，前沿模型与开源模型之间的一致性差距依然存在。

We position BCM as a process-level reliability signal that complements outcome metrics, and we are explicit about the conditions under which it is meaningful.

我们将 BCM 定位为一种补充结果指标的流程级可靠性信号，并明确了其发挥作用的适用条件。