SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

SearchAuditor：针对长程搜索智能体故障的审计与归因

Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. 深度搜索智能体通过长程网络交互来处理复杂问题，这一过程既复杂又脆弱：微小的推理错误可能会在漫长且充满噪声的轨迹中不断累积，最终导致虽然流畅但错误的答案。

Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. 诊断此类故障非常困难，需要人工检查极长的执行轨迹，这往往超出了人类的能力范围。因此，我们引入了 SearchAuditBench，这是一个旨在评估大模型（LLM）审计员能否定位、归因并修复这些故障的基准测试，从而减轻人工负担。

SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. SearchAuditBench 包含 1,243 条失败的执行轨迹，平均每条包含 73.1 条消息和 65.1K 个 Token。这些数据收集自五个深度搜索基准测试中的八个开源权重模型，每条轨迹都经过专家标注，明确了关键错误步骤、搜索特定的根本原因，并提供了带有评分标准的参考修复方案。

We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. 我们进一步提出了 SearchAuditor，这是一个多视角的审计框架，通过基于证据的裁决，有效地定位、归因并修复搜索智能体的故障。

Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors. 实验结果表明，即使是最强的基准模型，在由 GPT-5.5 等前沿模型驱动时，其端到端通过率也仅为 26.6%。相比之下，我们的 SearchAuditor 在不同前沿模型上始终优于所有基准模型，实现了 32.3% 的端到端通过率，并且利用其修复方案恢复失败的运行，能够使智能体更好地从错误中恢复。