Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark
Computer Science > Artificial Intelligence arXiv:2611.05459 (cs) [Submitted on 3 Nov 2026] Title: Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark Authors: Abdullah Al Kalbani, David M. Schultz, Alexandra K. Anderson-Frey, Zhonghua Zheng, Catherine McGuire.
计算机科学 > 人工智能 arXiv:2611.05459 (cs) [提交于 2026 年 11 月 3 日] 标题：大型语言模型能否审阅科学文献？针对 102 篇论文基准的测试 作者：Abdullah Al Kalbani, David M. Schultz, Alexandra K. Anderson-Frey, Zhonghua Zheng, Catherine McGuire。
Abstract: Large language models (LLMs) are increasingly used to support scientific literature reviews, yet evaluations often examine individual tasks rather than the review process as a whole. We assess six AI tools across four stages: finding relevant articles, interpreting them, synthesizing across them, and proposing future research.
摘要：大型语言模型（LLM）正越来越多地被用于辅助科学文献综述，但目前的评估往往只关注单一任务，而非整个综述过程。我们评估了六种人工智能工具在四个阶段的表现：查找相关文章、解读文章、综合分析以及提出未来研究方向。
The benchmark is a published review of the Spanish plume weather pattern and its 102-article corpus. In the initial searches, five tools collectively identified 57 articles, with the best individual tool finding 42. Institutional library access increased discovery from 42 to 82 articles for Codex and from 15 to 79 for Claude Code.
该基准测试基于一篇关于“西班牙羽流”（Spanish plume）天气模式的已发表综述及其包含 102 篇论文的语料库。在初步搜索中，五种工具总共识别出 57 篇文章，表现最好的工具找到了 42 篇。通过机构图书馆访问权限，Codex 的发现数量从 42 篇增加到 82 篇，Claude Code 则从 15 篇增加到 79 篇。
Given the complete corpus of 102 PDFs, three tools correctly identified all 18 articles discussing the Spanish plume substantially, but incorrectly classified 40% to 79% of the 84 passing mentions as substantial. The same three tools produced syntheses evaluated blindly by two domain experts.
在给定包含 102 份 PDF 的完整语料库的情况下，三种工具准确识别出了所有 18 篇实质性讨论“西班牙羽流”的文章，但错误地将 84 篇仅提及该主题的文章中的 40% 到 79% 归类为实质性讨论。这三种工具生成的综述由两位领域专家进行了盲审。
Although largely accurate, the syntheses provided limited insight and sometimes contained unsupported claims, quotation errors, and exaggerated statements of consensus. Finally, all three tools proposed research directions broader than the Spanish plume literature, although two approached questions identified in the benchmark review.
尽管综述在很大程度上是准确的，但它们提供的见解有限，有时包含未经证实的断言、引用错误以及对共识的夸大陈述。最后，这三种工具提出的研究方向都超出了“西班牙羽流”文献的范畴，尽管其中两种工具触及了基准综述中确定的问题。
No single tool performed best across all tasks. We provide eight recommendations for researchers using AI tools in literature reviews. Our findings suggest that LLMs can assist with discovering and interpreting scientific literature, particularly when institutional access is available, but cannot replace expert judgment in evaluating evidence or synthesizing research.
没有任何单一工具在所有任务中表现最佳。我们为在文献综述中使用人工智能工具的研究人员提供了八项建议。我们的研究结果表明，大型语言模型可以辅助发现和解读科学文献，特别是在拥有机构访问权限的情况下，但它们无法取代专家在评估证据或综合研究方面的判断。