Peerify: Benchmarking Peer-Review Claim Verification

Peerify：同行评审声明验证基准测试

Abstract: Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims.

摘要： 同行评审在学术出版中发挥着核心作用，然而，验证审稿人的声明是否得到手稿证据的支持，在很大程度上仍然是一个人工且耗时的过程。我们提出了 Peerify，这是一个用于基于手稿内容验证同行评审声明的流水线。

Given a manuscript and a review comment, the Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision.

给定一份手稿和一条评审意见，Peerify 流水线会将评审意见分解为原子声明，检索相关的手稿证据，并确定每一项声明是否得到论文的支持。为了支持该流水线的开发与评估，我们构建了一个包含 800 条声明的基准测试集，这些声明源自 NeurIPS 2024 和 ICLR 2024 的真实同行评审互动，其中包括一个包含 300 条声明的人工标注子集，用于审计自动化监督的结果。

We evaluate state-of-the-art language models and retrieval strategies within the Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims.

我们在 Peerify 流水线中评估了最先进的语言模型和检索策略，并结合了蕴含（entailment）基准模型。我们的结果证明了以检索为中心的验证和声明分解的重要性，同时也强调了模糊和解释性审稿人声明所带来的挑战。

Automated labels agree with human consensus on 90.3% of audited claims ($\kappa = 0.87$), while off-the-shelf entailment models stay below 0.24 macro-F1.

自动化标签在 90.3% 的审计声明中与人类共识达成一致（$\kappa = 0.87$），而现成的蕴含模型宏观 F1 分数则低于 0.24。