Inhibitory Attention for Clinical Long-Context Reasoning: Characterizing and Mitigating Lost-in-the-Middle Effects in EHR Processing

用于临床长上下文推理的抑制性注意力机制：表征并缓解电子健康记录（EHR）处理中的“中间丢失”效应

Abstract: Electronic health records now routinely exceed 100,000 tokens per patient. Yet large language models exhibit the lost-in-the-middle (LitM) effect: information near the center of a long context is retrieved less reliably than information near the edges. In clinical use this is not benign: the single most consequential fact in a note can sit at its center. We term this the clinical lost-in-the-middle (CLitM) problem, give its first systematic characterization using MedAlign, and compare context-selection strategies as remedies.

摘要： 电子健康记录（EHR）目前每位患者的数据量通常超过 100,000 个 token。然而，大型语言模型表现出“中间丢失”（Lost-in-the-Middle, LitM）效应：长上下文中靠近中心的信息比靠近边缘的信息检索可靠性更低。在临床应用中，这并非小事：病历中最关键的事实往往就位于中心位置。我们将此问题称为“临床中间丢失”（CLitM）问题，并利用 MedAlign 对其进行了首次系统性表征，同时对比了作为补救措施的上下文选择策略。

Across 2,196 instruction-response pairs and six language models, we observe a 21.9 percentage-point gap between peak accuracy (59.5%, 95% CI [46.3, 71.0], 20-30% decile) and trough accuracy (37.6% [23.2, 52.5] at 70-80%); 67.8% of reference answers fall between the 10th and 90th percentiles of the EHR timeline, inside the CLitM trough. We introduce Query-Conditioned Clinical Suppression (QCCS), a lightweight query-conditioned selection gate, and evaluate it against BM25, BM25 with section-header filtering, dense retrieval, and cross-encoder reranking (N=83 held-out instructions).

在 2,196 对指令-响应数据和六种语言模型中，我们观察到峰值准确率（59.5%，95% CI [46.3, 71.0]，位于 20-30% 分位区间）与谷值准确率（37.6% [23.2, 52.5]，位于 70-80% 分位区间）之间存在 21.9 个百分点的差距；67.8% 的参考答案落在 EHR 时间轴的第 10 到第 90 百分位之间，即处于 CLitM 的低谷区。我们引入了“查询条件临床抑制”（Query-Conditioned Clinical Suppression, QCCS），这是一种轻量级的查询条件选择门控机制，并将其与 BM25、带章节标题过滤的 BM25、稠密检索以及交叉编码器重排序（N=83 个留出指令）进行了对比评估。

With Qwen2.5-7B-Instruct (16k context), QCCS outperforms all five comparators under LLM-as-judge scoring: for middle-position instructions QCCS reaches 16.7% versus BM25 3.3%, cross-encoder 0.0%, dense 0.0%, and full context 6.7%; overall QCCS reaches 25.3% versus at most 3.6% for retrieval-only comparators. This advantage is not explained by retrieval recall: at k=20, BM25 retrieves the gold evidence sentence in 98.8% of instructions (QCCS 34.9%), yet retrieval arms stay at most 2.6% accurate even when they retrieve it, whereas QCCS reaches 25.0% even when it does not. In this proof-of-concept evaluation, query-aligned context selection predicts EHR instruction-following accuracy better than gold-sentence retrieval recall.

在使用 Qwen2.5-7B-Instruct（16k 上下文）的情况下，在“LLM 作为裁判”的评分标准下，QCCS 的表现优于所有五种对比方法：对于中间位置的指令，QCCS 达到了 16.7% 的准确率，而 BM25 为 3.3%，交叉编码器为 0.0%，稠密检索为 0.0%，全上下文为 6.7%；总体而言，QCCS 达到了 25.3% 的准确率，而仅检索类对比方法最高仅为 3.6%。这种优势无法用检索召回率来解释：在 k=20 时，BM25 在 98.8% 的指令中检索到了黄金证据句（QCCS 为 34.9%），但检索组即使检索到了证据，准确率最高也仅为 2.6%，而 QCCS 即使未检索到证据也能达到 25.0% 的准确率。在这一概念验证评估中，查询对齐的上下文选择比黄金句检索召回率更能预测 EHR 指令遵循的准确性。