SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators

SurveyReview：面向综述评估器的评审员对齐基准

The rapid advancement of large language models has transformed survey writing from a months-long manual effort into an automated process. As generation scales, reliable evaluation becomes the bottleneck, and LLMs are increasingly used as survey evaluators. However, existing approaches largely rely on off-the-shelf LLM-as-a-judge methods without systematic alignment to human reviewers, and there remains a lack of systematic frameworks for quantifying alignment with human reviewers.

大型语言模型的飞速发展已将综述写作从耗时数月的纯手工劳动转变为自动化流程。随着生成规模的扩大，可靠的评估成为了瓶颈，大模型也越来越多地被用作综述评估器。然而，现有的方法大多依赖于现成的“大模型作为裁判”（LLM-as-a-judge）模式，缺乏与人类评审员的系统性对齐，且目前仍缺乏量化这种对齐程度的系统性框架。

To address this gap, we propose SurveyReview, a reviewer-aligned, multi-dimensional benchmark and dataset for survey evaluation. We collect and annotate 675 survey papers with 1,630 review reports. We structure authentic peer-review reports by converting free-form comments into four-dimensional scores (Readability, Criticalness, Comprehensiveness, Structure) paired with supporting rationales. We further release standardized train/test splits and an evaluation protocol to measure alignment between automatic evaluators and human reviewers.

为了填补这一空白，我们提出了 SurveyReview，这是一个面向综述评估、与评审员对齐的多维度基准与数据集。我们收集并标注了 675 篇综述论文及其对应的 1,630 份评审报告。我们将自由形式的评审意见转化为四个维度的评分（可读性、批判性、全面性、结构性），并配以相应的论证依据，从而构建了结构化的真实同行评审报告。此外，我们还发布了标准化的训练/测试集划分以及评估协议，用于衡量自动评估器与人类评审员之间的对齐程度。

To validate the benchmark, we develop SurveyAlign, a strong baseline evaluator by fine-tuning Qwen3-32B with LoRA on our annotated data, augmented with external knowledge for knowledge-intensive dimensions. On the test set, SurveyAlign substantially improves reviewer alignment over prompt-based judging with GPT-5.2, reducing average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 across all four dimensions.

为了验证该基准，我们开发了 SurveyAlign，这是一个强大的基准评估器。它通过在我们的标注数据上使用 LoRA 微调 Qwen3-32B 模型，并针对知识密集型维度引入外部知识增强而成。在测试集上，SurveyAlign 在评审员对齐方面显著优于基于 GPT-5.2 的提示词评估方法，在所有四个维度上将平均均方误差（MSE）从 2.28 降低至 1.38，平均绝对误差（MAE）从 1.15 降低至 0.69。

Our contributions are twofold: (1) we establish the first multi-dimensional, reviewer-aligned dataset with a reproducible evaluation framework for survey reviewing; (2) we develop a strong baseline evaluator that substantially improves alignment with human reviewers, providing a competitive reference for future research. Our code and data are available at this https URL.

我们的贡献主要体现在两方面：（1）我们建立了首个多维度、与评审员对齐的数据集，并为综述评审提供了一个可复现的评估框架；（2）我们开发了一个强大的基准评估器，显著提升了与人类评审员的对齐效果，为未来的研究提供了具有竞争力的参考。我们的代码和数据已在指定链接公开。