RubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer Review

RubricReviewer：从直接批判到客观全面的评分标准驱动同行评审

Abstract: Peer review at major venues is under unprecedented submission pressure, motivating the use of large language models (LLMs) as review assistants. Existing LLM-based reviewers, however, face two structural limitations. First, they map manuscripts directly to reviews, leaving the underlying rubric implicit and entangling its derivation with the judgement. Second, the prevailing paradigms each capture only half of a good review: training-free agents gather broad evidence but produce undirected critiques, while training-based reviewers inherit human discriminative judgement together with its noise and uneven coverage.

摘要： 主要学术会议的同行评审正面临前所未有的投稿压力，这促使人们开始使用大语言模型（LLM）作为评审助手。然而，现有的基于 LLM 的评审系统面临两个结构性局限。首先，它们将手稿直接映射为评审意见，使得底层的评分标准（Rubric）处于隐性状态，并将标准的推导与最终判断混为一谈。其次，目前主流的范式往往只能实现高质量评审的一半功能：无需训练的智能体能够收集广泛的证据，但产生的批判缺乏针对性；而基于训练的评审员虽然继承了人类的判别能力，但也同时继承了其中的噪声和覆盖不均的问题。

We introduce RubricReviewer, a fully rubric-driven framework that addresses both limitations. It makes rubric generation an explicit intermediate step, so that both review generation and the final assessment are conditioned on paper-adaptive rubrics. It further combines a training-free agent (Scout) that gathers external evidence with a human-aligned trained model (Aligner) that consumes this evidence, fusing the strengths of both supervision sources.

我们引入了 RubricReviewer，这是一个完全由评分标准驱动的框架，旨在解决上述两个局限性。它将评分标准的生成作为一个明确的中间步骤，从而确保评审意见的生成和最终评估都基于针对论文自适应生成的评分标准。此外，该框架结合了一个无需训练的智能体（Scout，负责收集外部证据）和一个经过人类对齐训练的模型（Aligner，负责处理这些证据），从而融合了两种监督来源的优势。

Experiments on real-world submissions show that RubricReviewer produces reviews that are markedly more comprehensive and more discriminative than prior systems, and exhibits the strongest robustness against adversarial prompt-injection attacks. Ablation studies further confirm the necessity of each component.

在真实投稿上的实验表明，RubricReviewer 产生的评审意见比现有系统更加全面且更具判别力，并在对抗性提示注入攻击中表现出最强的鲁棒性。消融实验进一步证实了该框架中每个组件的必要性。