AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

Computer Science > Computation and Language arXiv:2610.22511 (cs) [Submitted on 15 Oct 2026] Title: AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems Authors: Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth.

计算机科学 > 计算与语言 arXiv:2610.22511 (cs) [提交于 2026 年 10 月 15 日] 标题：AuditKIT：用于基准测试、评判和比较大语言模型及智能体系统的统一评估骨干。作者：Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth。

Abstract: Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows.

摘要：评估语言模型和智能体需要结合基准准确性、基于评判的评估、运行时性能、工具使用行为以及检索质量。接口、指标语义和追踪格式的差异使这些比较变得复杂，并可能掩盖智能体工作流中的故障。

We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics.

我们提出了 AuditKIT，这是一个源码可用的评估库，它将这些测量结果记录在一种结果表示中，并具有方向感知的指标契约、配置指纹以及可扩展的指标和后端。对于智能体，它提供了轮次感知的工具调用匹配，区分了并行执行与顺序执行，并将已验证的任务结果与轨迹级诊断区分开来。

Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.

缺失或不可观察的证据被标记为未评分，而不是计为失败。对于检索增强生成，它结合了排序检索指标、评判依据和确定性证据检查。涵盖模型比较、服务运行时、检索、工具调用和互操作性的七个案例研究表明，答案提取、运行时行为和指标选择如何改变评估结果。