BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding

BrainBench：用于全面脑电图（EEG）理解的大语言模型基准测试

Abstract: Electroencephalography (EEG) analysis extends beyond assigning predefined labels to recordings; it requires workflows connecting natural-language instructions, signal processing, quantitative evidence, and scientific interpretation. We term this capability comprehensive EEG understanding. Existing evaluations, however, primarily target isolated decoding tasks or system-specific demonstrations, leaving the competence of large language models (LLMs) insufficiently quantified.

摘要： 脑电图（EEG）分析不仅仅是对记录分配预定义的标签；它需要连接自然语言指令、信号处理、定量证据和科学解释的工作流程。我们将这种能力称为“全面脑电图理解”。然而，现有的评估主要针对孤立的解码任务或特定系统的演示，导致大语言模型（LLMs）在这一领域的能力尚未得到充分量化。

We introduce BrainBench, a unified benchmark for comprehensive, instruction-conditioned EEG understanding. It comprises four subsets—Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration—covering 17 datasets, multiple tasks, and over 100,000 real-data instances. Given an instruction and EEG recordings with optional physiological signals, a system must perform the analysis and produce a scientifically grounded report and, when required, artifacts.

我们引入了 BrainBench，这是一个用于全面、指令驱动的脑电图理解的统一基准测试。它包含四个子集——基础分析、睡眠评估、神经认知评估和生理整合，涵盖了 17 个数据集、多个任务以及超过 10 万个真实数据实例。给定指令和包含可选生理信号的脑电图记录，系统必须执行分析并生成科学严谨的报告，并在需要时生成相关产物。

Outputs are assessed through numerical, categorical, set, sequence, semantic, and artifact validation. We evaluate representative LLMs across more than 100K executions under two paradigms: autonomous code execution with CodeAct and structured agentic analysis with BrainAgent. Results vary substantially across models, subsets, difficulty levels, and execution paradigms, showing that EEG competence depends on the model and its operationalization.

输出结果通过数值、分类、集合、序列、语义和产物验证进行评估。我们在两种范式下对代表性大语言模型进行了超过 10 万次的执行评估：使用 CodeAct 进行自主代码执行，以及使用 BrainAgent 进行结构化代理分析。结果在不同模型、子集、难度级别和执行范式之间存在显著差异，这表明脑电图处理能力取决于模型本身及其具体应用方式。

BrainBench provides a reproducible testbed for advancing LLM-based EEG understanding. The code and benchmark will be released soon, with evaluation results continuously updated.

BrainBench 为推进基于大语言模型的脑电图理解提供了一个可复现的测试平台。代码和基准测试即将发布，评估结果也将持续更新。