Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

智能体与工具交互中的“静默故障”：ToolUniverse 审计报告

Abstract: Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research and benchmarks have studied about task success and task completion of these agentic systems, the research about agent to tool interaction, specifically in biology agentic workflow is limited.

摘要： 智能体 AI 系统正越来越多地采用集成多种工具的自动化流水线。尽管先前的研究和基准测试已经探讨了这些智能体系统的任务成功率和任务完成情况，但关于智能体与工具交互（特别是在生物学智能体工作流中）的研究仍然有限。

This study investigates specific failures in agent to tool interaction where a tool invocation appears successful, some or all of the information or functionality from the tool via API/ wrapper is incomplete or missing and there are no communications / notifications to the user or the agent about such missing information. We call this a silent failures as the user or the agents are not aware that such failure has occurred.

本研究调查了智能体与工具交互中一种特定的故障：工具调用看似成功，但通过 API/封装器获取的部分或全部信息或功能不完整或缺失，且系统未向用户或智能体发送任何关于此类缺失信息的通信或通知。我们将此称为“静默故障”（Silent Failures），因为用户或智能体并未意识到此类故障已经发生。

For the purposes of this study we developed an audit mechanism to identify such silent failures in Agent to tool interaction, by examining 15 scientific tools (and their associated API documentation and tool documentations) integrated within ToolUniverse environment (ToolUniverse serves as our experimental environment rather than the object of the study itself). We structure our study around 7 failure locus characterising where the failure occurs in the chain.

为了进行本项研究，我们开发了一种审计机制，通过检查集成在 ToolUniverse 环境中的 15 种科学工具（及其相关的 API 文档和工具文档），来识别智能体与工具交互中的此类静默故障（ToolUniverse 仅作为我们的实验环境，而非研究对象本身）。我们将研究围绕 7 个故障位点展开，以表征故障在交互链条中发生的位置。

We observed 91 failures (manually validated post LLM based candidate discovery and automated testing), most frequent of them being missing data or fields and inconsistencies in search, filtering or ranking criteria. Most of the 91 failures occurred in API layer (51) or wrapper layer (25), with a potential of silent failure amplification downstream.

我们观察到了 91 起故障（在基于 LLM 的候选发现和自动化测试后进行了人工验证），其中最常见的是数据或字段缺失，以及搜索、过滤或排序标准的不一致。这 91 起故障中，大多数发生在 API 层（51 起）或封装层（25 起），并具有在下游放大静默故障的潜力。

The results show that silent failures originate upstream of the event and propagate downstream into apparently valid scientific outputs. We propose a concept of contextual reliability to handle such failures and suggest mechanisms for testing, disclosing, monitoring, and measuring such failures across the agent-tool interaction pipeline.

研究结果表明，静默故障起源于事件的上游，并向下游传播，最终形成看似有效的科学输出。我们提出了“上下文可靠性”（Contextual Reliability）的概念来处理此类故障，并建议在整个智能体-工具交互流水线中建立测试、披露、监控和衡量此类故障的机制。