Can we trust LLMs to judge whether a research idea is novel?

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Title: Can we trust LLMs to judge whether a research idea is novel?

标题：我们能相信大语言模型（LLM）来判断一个研究想法是否具有创新性吗？

Back to Articles Can we trust LLMs to judge whether a research idea is novel? Enterprise Article Published October 27, 2026 Upvote 6 Kyle Wiggers Ai2Comms Follow allenai 📄 Paper | 📊 Data | 💻 Code Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas.

回到文章列表：我们能相信大语言模型来判断一个研究想法是否具有创新性吗？企业文章，发布于2026年10月27日，获赞6次。作者：Kyle Wiggers，Ai2Comms，关注allenai。📄 论文 | 📊 数据 | 💻 代码。今天，我们发布了《旧想法，新问题》（Old Ideas, Novel Problems），这是一项关于大语言模型评估科学研究想法创新性可靠程度的新研究。

Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets. At Ai2, we’ve been developing AI systems that support different parts of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence across large bodies of literature into candidate scientific theories.

随论文一同发布的还有用于测试创新性评估器的基准测试、数据和代码，以及研究人员可用于构建新评估集的自动化流程。在Ai2，我们一直在开发支持科学发现不同环节的AI系统。DataVoyager帮助研究人员探索和分析结构化数据，AutoDiscovery生成并测试假设，而Theorizer则将大量文献中的证据综合为候选科学理论。

Together with Asta, these projects point toward systems that can help researchers move from understanding prior work to proposing, testing, and refining new ideas. But generating research ideas is only part of the challenge of automating scientific discovery. Those ideas also need to contribute something genuinely new, which means these systems need a way to distinguish a novel direction from one that mostly repackages existing work.

连同Asta项目，这些成果指向了能够帮助研究人员从理解前人工作过渡到提出、测试和完善新想法的系统。然而，生成研究想法只是自动化科学发现挑战的一部分。这些想法还需要做出真正的新贡献，这意味着这些系统需要一种方法来区分创新的方向与仅仅是重新包装现有工作的方向。

That judgment is often handed to another LLM—what we call a novelty judge. Novelty judges assess whether a research idea is new relative to prior work, either on its own or in comparison with another idea. There’s no standard way to build one; research teams choose a model, prompt, evaluation format, reasoning budget, and sometimes a literature-retrieval system. Those choices can materially affect the verdict.

这种判断通常交给另一个大语言模型——我们称之为“创新性评估器”。创新性评估器负责评估一个研究想法相对于前人工作是否具有新意，无论是独立评估还是与其他想法进行比较。目前还没有构建此类评估器的标准方法；研究团队需要选择模型、提示词、评估格式、推理预算，有时还需要选择文献检索系统。这些选择可能会实质性地影响评估结论。

Yet these judges are often validated only on human-written papers rather than the AI-generated ideas they are actually meant to evaluate. Our new benchmark results suggest novelty evaluation itself is far from reliable. Across six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.

然而，这些评估器通常仅在人类撰写的论文上进行验证，而非它们本应评估的AI生成想法。我们新的基准测试结果表明，创新性评估本身远未达到可靠的程度。在六种大语言模型评估器（GPT-5.1、GPT-5.2、GPT-5.4、Claude Sonnet 4.5、Claude Opus 4.5和Claude Opus 4.6）中，评估设置中看似微小的变化，都会导致对同一想法产生截然不同的判断。

In one experiment, changing just a few words in a prompt shifts a model’s accuracy in identifying which of two ideas is more novel by more than 50 percentage points. Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.

在一项实验中，仅修改提示词中的几个词，模型在识别两个想法中哪一个更具创新性时的准确率就会发生超过50个百分点的变化。检索和额外的推理并不能持续提高判断的准确性，而且两个专门构建的创新性评估系统表现甚至不如简单的Claude Opus 4.6提示词基准。

If AI is going to play a larger role in scientific discovery, we need more dependable ways to tell whether it is actually proposing something new. Our work provides tools for studying that problem, while showing how much remains unresolved. Building a benchmark for novelty judges To test how reliably novelty judges can distinguish more novel research ideas from less novel ones, we first needed examples where there was strong evidence about the originality of the underlying scientific contribution.

如果AI要在科学发现中发挥更大的作用，我们需要更可靠的方法来判断它是否真的提出了新东西。我们的工作为研究这一问题提供了工具，同时也展示了仍有多少问题尚未解决。构建创新性评估器的基准测试：为了测试创新性评估器区分研究想法创新程度的可靠性，我们首先需要一些关于底层科学贡献原创性有确凿证据的示例。

So we built an automated pipeline around ICLR 2026 peer reviews. Rather than treating paper acceptance as a proxy for novelty, we mined the reviews themselves for passages where reviewers explicitly discussed a contribution’s originality. That distinction is key; a paper can be accepted without being particularly novel as long as it is strong enough in other aspects. Likewise, a rejected paper with an original contribution can have other weaknesses.

因此，我们围绕ICLR 2026的同行评审构建了一个自动化流程。我们没有将论文录用作为创新性的代理指标，而是从评审意见中挖掘出评审员明确讨论贡献原创性的段落。这种区分至关重要；只要论文在其他方面足够出色，即使创新性不足也可能被录用。同样，一篇被拒的论文可能包含原创性贡献，但存在其他缺陷。

We created a high-novelty pool of 154 human-authored research ideas by selecting accepted papers in the top-rated 10% of their ICLR research area whose reviewers positively identified novelty and where no reviewer disputed it. We applied the reverse criteria to build a lower-novelty pool: 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns without any reviewer positively identifying novelty.

我们通过选择ICLR研究领域中评分排名前10%的录用论文，构建了一个包含154个人类撰写研究想法的高创新性池，这些论文的评审员明确肯定了其创新性，且没有评审员提出异议。我们应用相反的标准构建了一个低创新性池：从排名后10%的被拒论文中提取了145个想法，这些论文的评审员提出了原创性担忧，且没有任何评审员肯定其创新性。

In a separate check of the review-mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases; none of the disagreements would have changed whether a reviewer was counted as praising or disputing the paper’s novelty. Together, these two pools form our first evaluation setting, Human-Only, which tests whether novelty judges can tell apart human-authored ideas with strong evidence of higher or lower novelty.

在对评审挖掘过程的独立检查中，标注员在50个案例中有47个与提取出的创新性信号极性达成一致；没有任何分歧会改变评审员被归类为赞扬还是质疑论文创新性的结果。这两个池共同构成了我们的第一个评估设置“仅限人类”（Human-Only），用于测试创新性评估器能否区分具有高或低创新性确凿证据的人类撰写想法。

But in practice, novelty judges are increasingly being used to evaluate AI-generated ideas. So we created a second setting, Human+Generated. For this pool, we paired the same 154 validated human ideas with 154 ideas produced by a simple LLM generator. We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea.

但在实践中，创新性评估器正越来越多地被用于评估AI生成的想法。因此，我们创建了第二个设置“人类+生成”（Human+Generated）。对于这个池，我们将同样的154个经过验证的人类想法与154个由简单大语言模型生成器产生的想法进行了配对。我们假设生成的想法比人类撰写的想法创新性更低；这些是弱标签，而非对每个想法的验证性判断。

The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. This gave us a conservative comparison between ideas from a basic generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.

生成器仅接收研究领域和提出创新想法的指令，没有文献搜索、工具、迭代或反馈。这使我们能够在基础生成器产生的想法与评审员明确认可其原创性的ICLR前10%论文贡献之间进行保守的比较。

As additional checks, an expert independently reviewed 30 Human+Generated pairs, heavily weighted toward cases our strongest judge got wrong, and we rewrote the human- and AI-generated ideas into a common structure to test whether judges were simply reacting to differences in writing style. The expert preferred the human idea in all 30 pairs and judged none of the generated ideas novel.

作为额外的检查，一位专家独立审查了30对“人类+生成”配对，重点关注我们最强评估器判断错误的案例，并且我们将人类和AI生成的想法重写为统一结构，以测试评估器是否仅仅是对写作风格的差异做出了反应。专家在所有30对中都更倾向于人类的想法，并认为没有一个生成的想法具有创新性。

Importantly, many of our core findings don’t depend on the labels at all. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability. Small changes can produce very different verdicts Using our collections of human-authored and AI-generated research ideas, we tested how well six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – could evaluate novelty through both pointwise (i.e., absolute novelty ratings) and pairwise (i.e., comparing two idea

重要的是，我们的许多核心发现根本不依赖于这些标签。如果两个合理的评估设置对同一对想法给出了相反的答案，这种分歧本身就揭示了不稳定性。微小的变化可能产生截然不同的结论：使用我们收集的人类撰写和AI生成的研究想法，我们测试了六种大语言模型评估器（GPT-5.1、GPT-5.2、GPT-5.4、Claude Sonnet 4.5、Claude Opus 4.5和Claude Opus 4.6）通过点对点（即绝对创新性评分）和成对（即比较两个想法）评估创新性的能力。