AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

AgentLens：用于代码智能体评估的生产环境轨迹评估

We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit — did the task pass? — but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way.

我们提出了 AgentLens，这是一个针对交互式代码智能体的生产环境评估基准。大多数代码智能体基准测试将运行结果简化为一个单一的比特位——任务是否通过？——但实际使用这些智能体的用户体验的是整个交互轨迹：智能体如何遵循指令、使用工具、验证自身工作、从错误中恢复，以及在过程中如何与用户沟通。

AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is.

AgentLens 对整个轨迹进行评估。它将形式化验证（在存在客观检查标准的情况下）与大模型（LLM）编写的轨迹评估及并排对比相结合，从而使每次运行都能产生一份可读的解释，说明得分背后的原因。

This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at this https URL.

这使得 AgentLens 的用途不仅限于模型排名：我们利用它来诊断模型行为、比较我们自身智能体的连续版本，并在每日夜间评估流水线中捕捉产品回归问题。我们已将该基准测试开源，详情请访问此链接。