SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Computer Science > Artificial Intelligence arXiv:2611.05512 (cs) [Submitted on 3 Nov 2026] Title: SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories Authors: Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis.

计算机科学 > 人工智能 arXiv:2611.05512 (cs) [提交于 2026 年 11 月 3 日] 标题：SlackBench：基于真实代码仓库的协作项目智能体基准测试。作者：Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis。

Abstract: AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions.

摘要：人工智能智能体正逐渐成为数字同事，能够浏览网页、编写代码、运行实验并在长期范围内协助人类。然而，现有的基准测试无法满足实际知识工作的需求：许多测试依赖于合成数据或孤立的任务，忽略了协作项目中复杂的多方动态，而关键信息往往分散在对话、代码和不断演变的决策之中。

Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state.

即使是简单的工作场所问题，也可能需要定位相关的频道和私信、从冗长的讨论中恢复上下文、检查仓库状态、调和陈旧或冲突的证据，并跨来源进行综合分析。我们引入了 SlackBench，这是一个用于评估智能体在现实研究和工程工作流中表现的基准测试，这些工作流要求对工作场所沟通和不断演变的仓库状态进行联合推理。

Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations.

SlackBench 基于对真实学术研究项目的隐私保护重现，提供了可复现的环境，包含真实的对话、演变的决策以及基于 GitHub 仓库和 Google Docs 的相应代码变更。它包含 140 个查询，涵盖了异构来源的智能体搜索、对无法回答或前提错误问题的弃权、目标导向的摘要以及对长对话的情境推理。

These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 33 frontier models and agent harnesses, the best system scores around 80% and most agents score below 66%, highlighting substantial headroom for improvement.

这些任务不能简化为静态语料库问答：智能体必须确定哪些对话是重要的，遍历讨论串，检查仓库状态，并综合沟通和代码工件中的证据。在 33 个前沿模型和智能体框架中，表现最好的系统得分约为 80%，而大多数智能体的得分低于 66%，这凸显了巨大的改进空间。

SlackBench thus provides an extensible framework for evaluating agents in collaborative work environments. The SlackBench dataset is available at this https URL. Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL).

因此，SlackBench 为评估协作工作环境中的智能体提供了一个可扩展的框架。SlackBench 数据集可在该 https URL 获取。学科：人工智能 (cs.AI)；计算与语言 (cs.CL)。