GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2610.12617 (cs) [Submitted on 8 Oct 2026] Title: GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports Authors: Veronika Solopova, Ivan Havlytskyi, Viktoria Skorik, Ostap Vykhopen.

计算机科学 > 计算与语言 arXiv:2610.12617 (cs) [提交于 2026 年 10 月 8 日] 标题：GeopoliticalQA：针对情报报告的检索与生成基准测试作者：Veronika Solopova, Ivan Havlytskyi, Viktoria Skorik, Ostap Vykhopen。

Abstract: Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval-augmented question answering. We introduce a benchmark for grounded question answering over 415 open-source geopolitical and strategic intelligence reports, comprising 5,525 document-grounded question-answer pairs spanning factual, inferential, comparative, and risk-oriented questions.

摘要：地缘政治和战略情报报告是专家知识的丰富来源，在当前政治气候下具有日益增长的价值，但其异构性和分析复杂性为检索增强问答带来了挑战。我们引入了一个针对 415 份开源地缘政治和战略情报报告的扎根问答基准，包含 5,525 个基于文档的问答对，涵盖事实性、推理性、比较性和风险导向性问题。

We propose a document-aware generation pipeline that profiles reports before generating questions and validate dataset quality through expert annotation and an LLM-as-judge protocol. Using the benchmark, we evaluate 15 retrieval systems and six open-weight LLMs for RAG.

我们提出了一种文档感知生成流程，在生成问题之前对报告进行分析，并通过专家标注和“大模型作为裁判”（LLM-as-judge）协议验证数据集质量。利用该基准，我们评估了 15 个检索系统和 6 个开源权重的大语言模型在 RAG（检索增强生成）任务中的表现。

We find that models perform similarly on document-level retrieval, while at the harder chunk level retrieval quality is not driven by embedding-model scale: bge-m3, a mid-size open encoder matches an order of magnitude larger ones and outperforms commercial embedding APIs. In contrast, answer generation remains challenging even given the gold evidence, particularly for inferential and risk-oriented questions.

我们发现，模型在文档级检索上的表现相似，而在难度更高的分块级检索中，检索质量并不取决于嵌入模型的规模：中等规模的开源编码器 bge-m3 的表现与规模大一个数量级的模型相当，且优于商业嵌入 API。相比之下，即使提供了黄金证据，答案生成仍然具有挑战性，特别是在推理性问题和风险导向性问题上。