Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement

提取被迫流离失所与 FCV 文档中的数据集引用：一种基于大语言模型标签优化的弱监督框架

Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination.

发展与人道主义组织通过制作和支持调查、行政登记册及其他数据资源，为研究、政策制定和运营提供信息。然而，系统性地识别这些数据集在何处被引用仍然十分困难。此类引用分散在研究论文、项目文档、人道主义报告及其他非结构化文本中，这不仅限制了追踪数据使用的能力，也阻碍了识别数据可用性或传播中潜在缺口的工作。

We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction.

我们提出了一种弱监督框架，旨在将数据集提取技术应用于被迫流离失所及“脆弱、冲突与暴力”（FCV）相关文档，且无需预先构建大规模的人工标注训练语料库。该方法利用在通用研究文献上训练的轻量级模型，从无标签的领域文档中生成候选数据集引用；随后，由前沿大语言模型（LLM）在语境中进行审查，对候选内容进行验证或剔除，并修正其提取边界。最终生成的标注数据辅以针对性的合成示例与对比示例，用于微调轻量级模型，从而实现大规模提取。

We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1% precision and 70.5% recall at the mention level; among passages containing dataset references, precision reaches 89.5%. At the passage level, the model achieves 88.2% accuracy and 88.6% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.

我们在一个包含 1,706 个文本段落的独立黄金标准基准上评估了该模型，这些段落涵盖了研究、人道主义及运营文档。在整个基准测试中，该模型在引用层面的精确率为 74.1%，召回率为 70.5%；在包含数据集引用的段落中，精确率达到了 89.5%。在段落层面，模型在区分包含与不包含数据集引用的段落时，达到了 88.2% 的准确率和 88.6% 的特异性。这些结果证明了一种在标注数据有限的情况下构建领域特定监督的实用方法，并为更大规模地分析流离失所数据领域中的数据使用情况及潜在缺口提供了技术基础。