Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss

基于机构特定提示词的大语言模型可找回现有去标识化系统及其“黄金标准”所遗漏的受保护健康信息 (PHI)

Abstract: Secondary use of electronic health records requires de-identification, yet existing systems miss institutionally situated protected health information (PHI) such as hospital abbreviations, building names, and internal codes whose status is locally determined. We ask whether large language models (LLMs) with in-context learning (ICL) can close this gap and control the precision—recall trade-off.

摘要： 电子健康记录的二次利用需要进行去标识化处理，然而现有系统往往会遗漏那些“机构特定”的受保护健康信息 (PHI)，例如医院缩写、建筑物名称以及由本地定义的内部代码。我们探讨了利用上下文学习 (ICL) 的大语言模型 (LLM) 是否能够弥补这一差距，并控制精确率与召回率之间的权衡。

On 100 annotated pediatric oncology notes (5,322 PHI spans) from Texas Children’s Hospital, we benchmarked eight LLMs against two purpose-built systems (Stanford TiDE, OpenMed PII) and two pattern-based baselines. Each LLM ran under three prompts of increasing specificity: (1) a HIPAA-aligned baseline, (2) baseline plus the institutional PHI categories it missed, and (3) prompt 2 plus instructions against over-redacting clinical content. We then compared 14 multi-agent and ensemble configurations against the best single prompt, with recall the primary safety metric.

我们在德克萨斯州儿童医院的 100 份标注儿科肿瘤病历（包含 5,322 个 PHI 片段）上，将 8 个大语言模型与两个专用系统（Stanford TiDE、OpenMed PII）及两个基于模式的基准系统进行了对比测试。每个大语言模型在三种特异性递增的提示词下运行：(1) 符合 HIPAA 标准的基准提示词；(2) 基准提示词加上模型此前遗漏的机构特定 PHI 类别；(3) 在第 2 种提示词基础上增加防止过度遮盖临床内容的指令。随后，我们将 14 种多智能体和集成配置与最佳的单一提示词进行了比较，并将召回率作为主要的安全性指标。

LLMs outperformed the purpose-built systems (best F1=0.918±0.001 vs. TiDE 0.779), with advantages concentrated in contextual categories. Naming the missed categories recovered 79% (48/61) of them, and discouraging over-redaction restored precision. No agentic architecture beat calibrated single-pass prompting (F1 0.906–0.907), but LLM outputs surfaced 414 candidate annotation gaps; re-annotation confirmed 227 PHI spans, against which the final prompt reached recall=0.981 (F1=0.907±0.002).

大语言模型的表现优于专用系统（最佳 F1 分数为 0.918±0.001，而 TiDE 为 0.779），其优势主要集中在上下文相关的类别中。明确指出遗漏的类别后，模型找回了其中 79% (48/61) 的信息，而通过抑制过度遮盖则恢复了精确率。没有任何智能体架构的表现超过了经过校准的单次提示词处理（F1 分数为 0.906–0.907），但大语言模型的输出揭示了 414 个潜在的标注缺失；重新标注确认了其中的 227 个 PHI 片段，基于此，最终提示词的召回率达到了 0.981 (F1=0.907±0.002)。

Well-calibrated ICL resolves both the institutional PHI gap and the precision—recall trade-off in one LLM call per note. LLMs cost more to run than traditional methods, but that cost buys a way to audit the reference standard. LLMs are a legitimate, adaptable alternative to purpose-built de-identification systems; institution-specific prompt development should be the primary adaptation strategy.

经过良好校准的上下文学习 (ICL) 可以在每份病历仅需一次大语言模型调用的情况下，同时解决机构特定 PHI 的遗漏问题以及精确率与召回率的权衡问题。虽然大语言模型的运行成本高于传统方法，但这一成本换取了审计参考标准的能力。大语言模型是专用去标识化系统的一种合法且灵活的替代方案；针对机构特定情况开发提示词应成为主要的适配策略。