TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.30161 (cs) [Submitted on 30 Sep 2026] Title: TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents Authors: Krishanu Adhikary, Prashant Devadiga.

计算机科学 > 计算与语言 arXiv:2610.30161 (cs) [提交于 2026 年 9 月 30 日] 标题：TauIndianBankBench：面向工具使用型对话智能体的流程感知与行动导向评估作者：Krishanu Adhikary, Prashant Devadiga。

Abstract: Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank’s rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit.

摘要：许多印度银行通过常见问题解答页面和脚本化聊天机器人为客户提供服务。基于语言模型的助手可以更快地响应并执行诸如冻结卡片之类的请求，但前提是它们必须提供正确的信息并遵守银行的规则。这些规则规定了流程，即一系列必要的步骤，例如在关闭存款账户前获取结清报价。

Adapting models to this domain requires measuring how well they follow the procedure, before and after training. Benchmarks usually judge only the records left at the end of a conversation. Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful.

将模型应用于该领域需要衡量它们在训练前后遵循流程的程度。基准测试通常仅评估对话结束时留下的记录。然而，许多必要的步骤并不会改变记录，因此跳过这些步骤的助手仍可能被判定为成功。

We present TauIndianBankBench, an action-oriented benchmark of 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. Besides the final records, it checks that the assistant took every required step and made no unrequested change.

我们提出了 TauIndianBankBench，这是一个包含 1,000 项印度零售银行业务任务的行动导向基准测试，其中助手会协助模拟客户。除了最终记录外，它还会检查助手是否执行了每一个必要步骤，且未进行任何未经请求的更改。

For every model, 6.5-12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change. Scoring both procedure and outcome shows more accurately how well language models perform in Indian retail banking.

对于每个模型，当仅检查最终记录时，有 6.5% 到 12.0% 的会话被判定为成功，尽管助手实际上跳过了必要步骤或进行了未经请求的更改。同时对流程和结果进行评分，能更准确地反映语言模型在印度零售银行业务中的表现。