Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs
Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs
利用微调大语言模型识别英国警方事件记录中的脆弱性指标
Abstract: 摘要：
Purpose: Understanding how much of routine policing involves vulnerable people could inform resourcing, training, and multi-agency response, yet administrative data provide limited insight. We explore whether an LLM-based classification pipeline, developed on open-source US police data, can be adapted to estimate the prevalence of four vulnerability indicators - mental ill health, substance misuse, alcohol dependence, and homelessness - in UK police incident narratives, and when outputs can be treated as defensible measurements. 目的：了解日常警务工作中涉及脆弱人群的比例，有助于优化资源配置、培训及多机构协作响应，但目前的行政数据提供的见解有限。我们探讨了基于开源美国警务数据开发的大语言模型（LLM）分类流水线，是否可以调整用于估算英国警方事件记录中四种脆弱性指标（心理健康问题、药物滥用、酒精依赖和无家可归）的普遍程度，并研究在何种情况下其输出结果可被视为可靠的衡量标准。
Methods: We analyse nearly 3,000 de-identified incident logs from a UK police force, using a multi-stage pipeline combining repeated model inference, label aggregation, structured human review, and statistical correction. The pipeline runs on a locally hosted open-weight LLM, reflecting the secure environments police must work in. 方法：我们分析了来自英国某警方的近 3,000 条去标识化事件记录，使用了一个结合了重复模型推理、标签聚合、结构化人工审核和统计校正的多阶段流水线。该流水线运行在本地托管的开放权重 LLM 上，以符合警方必须工作的安全环境要求。
Results: LLMs can produce meaningful, if imperfect, prevalence estimates at scale. Mental ill health indicators are present in approximately one in five incidents, with lower prevalence for other indicators. However, naive LLM deployment is unreliable: single-pass classifications are unstable, and aggregated outputs systematically over-assign indicators relative to human judgement. Correcting these biases required substantial human input and statistical adjustment, leaving considerable uncertainty. 结果：大语言模型能够在大规模范围内产生有意义但并不完美的流行度估算。心理健康问题指标出现在约五分之一的事件中，其他指标的出现率较低。然而，直接部署 LLM 是不可靠的：单次分类结果不稳定，且聚合后的输出相对于人工判断而言，系统性地过度分配了指标。纠正这些偏差需要大量的人工投入和统计调整，且仍存在相当大的不确定性。
Conclusions: While LLMs can extract information from unstructured police data, their outputs cannot be treated as valid measurements without careful methodological support. At the population level, defensible estimates are achievable but resource-intensive; at the individual level, errors remain frequent and unpredictable, limiting suitability for operational decisions. This study highlights both the potential and the constraints of LLM-based measurement in applied settings. 结论：虽然大语言模型可以从非结构化的警务数据中提取信息，但在没有严谨方法论支持的情况下，其输出不能被视为有效的衡量标准。在群体层面，可以获得可靠的估算，但资源消耗巨大；在个体层面，错误依然频繁且不可预测，限制了其在业务决策中的适用性。本研究强调了基于 LLM 的测量在应用场景中的潜力和局限性。