Your Agent Aced the Task. Will It Do It Again?
Your Agent Aced the Task. Will It Do It Again?
你的智能体出色地完成了任务。它还能再做一次吗？
Your agent works in rehearsal, but during the live demo, it takes a different path and fails the same task. That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper. 你的智能体在排练时表现良好，但在现场演示时却走了另一条路，导致任务失败。这在舞台上非常尴尬。在生产环境中，这是一个可靠性问题：一个曾经成功的工作流，在用户下次提出相同请求时可能会失败。对于关键任务，例如核对财务交易或检查合同义务，这可能是致命的阻碍。
Most benchmarks hide this variability behind an average. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions. But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap. Most benchmarks report the first number. We built a way to measure the second — and improve it. 大多数基准测试通过平均值掩盖了这种变异性。在 AppWorld 上，一个使用 GPT-4.1 的 ReAct 智能体在五次重复运行中，有 77.4% 的运行获得了成功。但只有 53.0% 的任务在五次运行中全部成功——存在 24.4 个百分点的“一致性差距”。大多数基准测试只报告第一个数字。我们构建了一种衡量第二个数字的方法，并对其进行了改进。
In an earlier post, we introduced ALTK-Evolve — a system that turns an agent’s own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. It measurably improves task success, but those results only asked the average-case question too. This post introduces consistency guidelines, a new guideline type in altk-evolve built on top of a diagnostic tool we call the Consistency Analyzer, that targets this gap directly. 在之前的一篇文章中，我们介绍了 ALTK-Evolve——一个将智能体自身过去的轨迹转化为可重用指南的系统，这些指南会自动提炼并在推理时注入。它显著提高了任务成功率，但这些结果也只是回答了平均情况下的问题。本文介绍了“一致性指南”（consistency guidelines），这是 ALTK-Evolve 中的一种新指南类型，它基于我们称为“一致性分析器”（Consistency Analyzer）的诊断工具，直接针对这一差距。
TL;DR Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld test_normal) that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks — a 24.4-point consistency gap. On hard tasks it reaches 30 points. We built a diagnostic for exactly this. The Consistency Analyzer resamples an agent’s own recorded trajectory to find flip-prone decision points — steps where the model was one token-sample away from doing something different. It needs one trace and no ground truth — it resamples each decision point in that trace with a single call requesting k completions (k=5 by default), rather than re-running the task end-to-end. Turning that diagnosis into guidelines halves the gap — from 24.4pp to 12.0pp (same-task Pass⁵ +16.0pp, similar-task +13.0pp), without costing anything in average accuracy. Full methodology and evaluations are in the technical report on arXiv. 简而言之：准确率掩盖了不可靠的问题。一个平均成功率为 77.4% 的 ReAct 智能体（在 AppWorld test_normal 上使用 GPT-4.1），在 5 次重复运行中只有 53.0% 的任务能全部成功——存在 24.4 个百分点的一致性差距。在困难任务中，这一差距甚至达到 30 个百分点。我们专门为此构建了一个诊断工具。一致性分析器通过重采样智能体自身记录的轨迹，找出容易发生“翻转”的决策点——即模型距离做出不同选择仅差一个 Token 采样的步骤。它只需要一条轨迹且无需标准答案——它通过单次调用请求 k 个补全（默认 k=5）来重采样轨迹中的每个决策点，而不是端到端地重新运行任务。将该诊断转化为指南可将差距减半——从 24.4pp 降至 12.0pp（同任务 Pass⁵ 提升 16.0pp，相似任务提升 13.0pp），且不会牺牲平均准确率。完整的方法论和评估报告可在 arXiv 上查阅。
The Metric Almost Nobody Reports
几乎没人报告的指标
Standard agent evaluation reports Mean@k: run a benchmark k times, average the pass rate. Often k=3, sometimes just 1. It’s the number on every leaderboard, and it’s what “77% accurate” means in practice. Mean@k answers “how good is this agent, on average?” It does not answer the question a real user cares about: will it still be good if I ask this exact question again? For that you need Pass^k: the fraction of tasks where the agent succeeds on all k runs. 标准的智能体评估报告使用 Mean@k：运行基准测试 k 次，取通过率的平均值。通常 k=3，有时仅为 1。这是每个排行榜上的数字，也是“77% 准确”在实践中的含义。Mean@k 回答的是“这个智能体平均表现如何？”它并没有回答真实用户关心的问题：如果我再次提出完全相同的问题，它还能表现良好吗？为此，你需要 Pass^k：即智能体在所有 k 次运行中均成功的任务比例。
⚠️ Pass^k is not Pass@k. The familiar Pass@k is optimistic — it asks whether at least one of k attempts succeeded, the right question when you can verify and retry. Pass^k is its pessimistic mirror image: every attempt must succeed. Same letters, opposite question. Pass^k ≤ Mean@k ≤ Pass@k, always. ⚠️ Pass^k 不等于 Pass@k。我们熟悉的 Pass@k 是乐观的——它询问 k 次尝试中是否有至少一次成功，这在你可以验证并重试的情况下是正确的问题。Pass^k 是其悲观的镜像：每一次尝试都必须成功。字母相同，问题相反。始终满足 Pass^k ≤ Mean@k ≤ Pass@k。
A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4% — genuinely strong. But Pass^5 is only 53.0%. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes can’t, with nothing about the task changing between runs. We call this gap — Mean@k minus Pass^k — the consistency gap. This isn’t a capability problem you fix with a bigger model. It’s an orthogonal axis: an agent can be capable and inconsistent at the same time. 一个由 GPT-4.1 支持的 ReAct 智能体 Mean@5 为 77.4%——表现确实很强。但 Pass^5 仅为 53.0%。近四分之一的基准测试任务属于智能体有时能解决、有时不能解决的情况，而任务本身在运行之间没有任何变化。我们将这种差距（Mean@k 减去 Pass^k）称为“一致性差距”。这不是一个可以通过使用更大模型来解决的能力问题。这是一个正交的维度：智能体可能既有能力又不一致。
Why Agents Flip: Sharp Decisions vs. Flat Ones
为什么智能体会“翻转”：尖锐决策与平坦决策
Every time an LLM agent decides something — which API to call, what argument to pass, whether to retry — that decision comes out of a probability distribution over next tokens. What matters is the shape of that distribution. A sharp one puts most of its mass on a single token: the runners-up are far behind, and the same choice comes out run after run. A flat one spreads comparable mass across several near-tied tokens, and which one wins is close to a coin flip. The shape decides how much noise it takes to change the outcome. 每当 LLM 智能体做出决定时——调用哪个 API、传递什么参数、是否重试——该决定都源于下一个 Token 的概率分布。重要的是该分布的形状。一个“尖锐”的分布将大部分权重集中在单个 Token 上：次优选项远远落后，因此每次运行都会做出相同的选择。一个“平坦”的分布将相当的权重分散在几个接近的 Token 上，谁胜出就像掷硬币一样。分布的形状决定了需要多少噪声才能改变结果。
Sharp distributions are resilient — GPU floating-point non-associativity, request batching, and other platform-side effects nudge the numbers slightly, but nowhere near enough to reorder a clear winner. Flat distributions are vulnerable to exactly that nudge: near-ties may reorder under small perturbations. And because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. That’s where a 24-point gap comes from. 尖锐的分布具有弹性——GPU 浮点运算的非结合性、请求批处理以及其他平台侧效应会轻微扰动数值，但不足以改变明显的胜出者。平坦的分布则极易受到这种扰动的影响：在微小扰动下，接近的选项可能会重新排序。由于轨迹由数十个决策串联而成，每一步微小的“翻转”概率会累积成某次运行结果不同的巨大可能性。这就是 24 个百分点差距的来源。
This is also why the problem survives your decoding settings. Greedy decoding and a fixed seed both govern how a distribution gets turned into a token — they say nothing about the distribution itself. On a hosted endpoint the probabilities shift slightly from run to run, so the same prompt to the same model at temperature zero can still resolve a near-tie one way today and the other way tomorrow. Our setup: the ReAct agent runs at temperature 0.0, so none of the variance above is ordinary sampling. 这也是为什么这个问题在你的解码设置下依然存在的原因。贪婪解码和固定种子都只决定了分布如何转化为 Token——它们对分布本身没有任何影响。在托管端点上，概率在每次运行之间会略有偏移，因此即使在温度为 0 时，同一个提示词发给同一个模型，今天可能以一种方式解决接近的选项，明天可能以另一种方式解决。我们的设置是：ReAct 智能体在温度 0.0 下运行，因此上述变异均非普通的采样所致。
Diagnose, Then Fix
先诊断，后修复
Which turns the problem into a search: which steps in a given trajectory were the flat ones — and what do you do about them once you know? Consistency guidelines come out of a two-stage pipeline that plugs into ALTK-Evolve’s existing machinery — with a new source signal driving what gets written. 1. Detect — the Consisten… 这使得问题转化为一个搜索过程：给定轨迹中的哪些步骤是“平坦”的——一旦知道，你该如何处理它们？一致性指南源于一个两阶段流水线，它插入到 ALTK-Evolve 现有的机制中——并由一个新的源信号驱动写入内容。1. 检测——一致性…