ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Back to Articles ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model Enterprise Article Published October 20, 2026 Upvote 20 +14 Ashwath Vaithinathan Aravindan ashwath-vaithina Follow ibm-research Vatche Isahagian Vatche Follow ibm-research Vinod Muthusamy vinodmut Follow ibm-research Jayaram Radhakrishnan jayaramkr Follow ibm-research Evelyn Duesterwald evduester Follow ibm-research Gaodan Fang gaodan-fang Follow ibm-research Punleuk Oum illeatmyhat Follow ibm-research G Thomas gsthomasx Follow ibm-research Michał Ulewicz Michal Follow ibm-research Merve Unuvar mrvnvr Follow ibm-research Ayhan Sebin ayhansebin Follow ibm-research ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights.

ALTK-Evolve 通过使用 GPT-4.1 学习并应用一套“剧本”（playbook），在客户服务代理基准测试 τ²-bench 上超越了零售类参赛作品，且无需更新模型权重。

Learned memory takes a basic ReAct agent with GPT-4.1 to the top Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches 67.5% Pass^4 on τ²-bench retail — meaning it solves the task on all four independent attempts for 67.5% of tasks. It scores above every retail entry in our October 12, 2026 leaderboard snapshot. It runs on the benchmark’s stock ReAct scaffold, with learned guidelines appended to its system prompt.

习得的记忆将一个基于 GPT-4.1 的基础 ReAct 代理推向了榜首。使用较旧的非推理模型 GPT-4.1，一个基础客户服务代理在 τ²-bench 零售测试中达到了 67.5% 的 Pass^4 率——这意味着它在 67.5% 的任务中，四次独立尝试均能成功解决问题。在 2026 年 10 月 12 日的排行榜快照中，它的得分超过了所有零售类参赛作品。它运行在基准测试自带的 ReAct 框架上，并将习得的指导原则附加到其系统提示词中。

ALTK-Evolve builds that playbook using GPT-4.1. It analyzes the agent’s training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent’s prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed. Our previous work demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer’s request, and executing the right tool actions.

ALTK-Evolve 使用 GPT-4.1 构建了这套剧本。它分析代理在训练任务中的尝试，并利用该模型将成功和失败的经验提炼为简短的行为规则：例如如何验证客户身份、确认交易或处理退货。随后，这些指导原则被添加到代理的提示词中，用于后续对话。同一个模型既负责提取指导原则，也负责驱动代理；无需进行权重更新或人工标注。我们之前的工作已在编码代理上验证了这种方法。在这里，我们将其应用于客户服务领域，在该领域，成功需要遵循领域政策、理解客户需求并执行正确的工具操作。

τ²-bench tests policy-bound customer service τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward.

τ²-bench 测试受政策约束的客户服务。τ²-bench（来自 Sierra）旨在衡量代理在真实约束条件下处理客户服务对话的能力。每个任务都将代理与一名具有特定目标的 LLM 模拟客户配对，并提供一套领域工具（如查询订单、发放退款、重新预订航班）以及代理必须遵守的书面政策文档。代理必须提取正确的信息、保持在政策范围内并采取正确（通常是不可逆的）行动；只有当最终数据库状态与预期结果匹配时，任务才算解决，这是一种严格的二元奖励机制。

We use its two customer-service domains, retail and airline. We run τ²-bench’s own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model; 4 trials per task. The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list: You are a customer service agent that helps users according to the policy below. Always follow the policy strictly. Use the provided tools when needed.

我们使用了其零售和航空两个客户服务领域。我们运行了 τ²-bench 自带的基础 ReAct 代理实现，该实现包含一个两阶段的“思考 → 行动”（THINK → ACT）循环，使用 gpt-4.1 作为代理模型，gpt-5.5 作为客户模拟模型；每个任务进行 4 次试验。我们所做的唯一改变是在系统提示词后附加了一块习得的指导原则——其他所有内容（政策、工具、框架）均保持为 τ²-bench 的原始设置。具体而言，习得的指导原则以纯 Markdown 列表的形式附加在领域政策之后：你是一名客户服务代理，根据以下政策帮助用户。请务必严格遵守政策。在需要时使用提供的工具。

# Retail agent policy As a retail agent, you can help users: - cancel or modify pending orders - return or exchange delivered orders - modify their default user address … ## Guidelines learned from past task attempts (follow these carefully): - Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API. - Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action. - When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don’t wait for the user to ask. - Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool’s result before replying. - Before looking a user up, parse and clean their name — e.g. split a combined handle like daiki_johnson_9523 into first/last name — and after a couple of failed lookups, state the exact format you need. …

# 零售代理政策 作为零售代理，你可以帮助用户： - 取消或修改待处理订单 - 退货或换货已送达订单 - 修改默认用户地址 … ## 从过往任务尝试中习得的指导原则（请仔细遵循）： - 在采取任何与订单相关的操作前，先验证用户身份。如果无法获取电子邮件，请使用 find_user_id_by_name_zip API 通过全名和邮政编码验证身份。 - 在执行任何数据库更新操作前，务必通过清晰、结构化的摘要确认所有拟进行的更改（修改、取消、退货、换货），并要求用户明确批准。 - 在处理退货或换货时，请提供包含明细的退款说明并明确退款时间——不要等用户询问。 - 每步仅执行一个操作：切勿在同一轮对话中将工具调用与面向用户的消息合并，并等待工具返回结果后再回复。 - 在查询用户前，解析并清理其姓名——例如，将像 daiki_johnson_9523 这样的组合句柄拆分为名和姓——如果多次查询失败，请说明你需要的确切格式。 …

We extract guidelines from the agent’s trajectories on τ²-bench train tasks using gpt-4.1, the same model that powers the agent, then apply them unchanged on the held-out test set. Guidelines improve capability and reliability In a prior post, we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model. Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off.

我们使用驱动代理的同一个模型 gpt-4.1，从代理在 τ²-bench 训练任务的轨迹中提取指导原则，然后将其原封不动地应用于留出的测试集。指导原则提高了能力和可靠性。在之前的一篇文章中，我们讨论了不同的模型/任务需要不同剂量的指导。基于这一理念，我们也评估了提供给模型的不同数量的指导原则。每条指导原则都带有一个支持计数——即它从多少个不同的训练任务中提取出来，因此更高的支持度意味着该规则在更多任务中具有通用性，而非一次性规则。

We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e. support ≥ 1). Table 1. Guideline configurations and retained guideline counts by domain. Variant Guidelines included in context Retail Airline baseline no guidelines 0 0 sup3 support n ≥ 3 (high-support core) 31 14 lossless all guidelines 292 101

我们比较了三种配置：基准（无指导原则）、sup3（仅保留支持度 ≥ 3 的指导原则——一个小型、高置信度的核心）和无损（保留整合后留下的所有指导原则，即支持度 ≥ 1）。表 1. 各领域的指导原则配置及保留的指导原则数量。变体 上下文中包含的指导原则 零售 航空 基准 无指导原则 0 0 sup3 支持度 n ≥ 3（高支持度核心） 31 14 无损 所有指导原则 292 101

The metrics we track. Every task is attempted 4 times, and we read those trials two ways: Mean Accuracy — the average success rate across the 4 trials. This is the capability axis: how often does the agent get this right? Pass^4 — the percentage of tasks that passed across all 4 trials. This measures along the reliability axis and the metric the τ²-bench leaderboard reports. It matters here because a support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers — Pass^4 asks whether you can trust the agent every time, not j

我们追踪的指标。每个任务尝试 4 次，我们通过两种方式解读这些试验：平均准确率——4 次试验中的平均成功率。这是能力维度：代理正确完成任务的频率如何？Pass^4——在所有 4 次试验中均通过的任务百分比。这是衡量可靠性的维度，也是 τ²-bench 排行榜报告的指标。这一点很重要，因为一个在 4 次退款请求中能解决 3 次的客服代理，仍然会让四分之一的真实客户失望——Pass^4 衡量的是你是否能每次都信任该代理，而不是 j